Audit logging with Kinesis Firehose

Audit logs

We had a requirement to maintain an audit log of user actions that management would be able to report on one day. Our systems currently forward logs to Datadog, but we have fairly short retention periods. Extending the retention period for a limited number of logs would be too costly at this stage.

The other option that seemed to make sense was storing these logs in S3 - they could be easily search at some stage later using AWS Athena. I'd heard of Kinesis Firehose before, so this seemed to be an ideal case to try it out.

Kinesis Firehose allows you to put events into the stream and configure an output destination, of which S3 is one. It takes care of bundling events into files, transforming and outputting the data at very little cost, all without standing up any servers.

The code

To setup Kinesis Firehose through cloudformation (Serverless specifically) you can use the following configuration and code.

Creating the infrastructure is really fast, so if you create a new stack for each branch of code, this will work well with your pattern. If you put your branch into the s3 prefix, it'll make life easier to determine which logs belong to which branch, and you can run Athena based on a S3 prefix, not just the whole bucket

Stream Configuration

Create an IAM role for the stream that allows it to write to S3 in a given path, configure the stream

Push events in our app

Now that we have a stream, it's simple to create a module for sending events:

JavaScript filter thisArg

I was attempting to write an amazing filter that would just return an array of objects where a property was true or false (and I wanted to be able to choose a value at runtime). This seemed reasonably straight forward:

function filterBySomeProp(object) {
  return object.someBoolProp === this;
}
bigArrayOfObjects.filter(filterBySomeProp, true);

Unfortunately it didn't give the expected result - turns out when passing the primitive bool as the thisArg, it gets converted to a Boolean object, and strict equality checks fail (e.g '===') as the value and type is checked. This is fixed by wrapping the primitive value when doing the check:

function filterBySomeProp(object) {
  return Boolean(object.someBoolProp) === this;

}

Happy filtering!

Learning in the open - mobprogramming

One of the best things about mob programming (#mobprogramming on twitter) is the way that the team shares learning.

Our team recently worked on a tool to remove unused AWS stacks, dedicating quite a few lunchtimes to the cause. We decided it would be a great opportunity to try some mob programming, as it was a project that we were all excited about.

In the process we

• Learnt some emacs
• Learnt about using the Serverless framework (AWS lambda under the hood) for the first time
• Worked out how we were going to test our application
• Familiarised ourselves with recursion and promises in javascript
• Came up against deployment and runtime problems

While each of these could have been done solo, mob programming gave us the opportunity to:

• Talk the problems through, coming up with better design
• Get comfortable asking for explaination when we're unclear - showing that we don't know it all
• Bring something new to the table that other people don't know - even if we've just found it ourselves
• Teach others
• Share the joy of seeing something working that we've worked on so closely together

One of the biggest benefits is individuals showing they don't know everything. Often as software developers we suffer from imposter syndrome, but just exposing that we're all learning can encourage a better culture where people are happy being vulnerable - where they feel they can ask questions without judgement.

Have you tried mob programming? What do you think are the benefits? Drawbacks?

Spinning down your cloud costs

One of the best things about the cloud is the ability to spin up new infrastructure for your dev environment. You can have a completely isolated environment for a new branch that you're working on, allowing you to test your changes in isolation.

This is fantastic as you can speed ahead on changes unencumbered - the only problem is that you'll leave a trail of cash sitting in your cloud providers pockets, which often doesn't work out so well for your business.

To attack our cost blowouts, we started a side project in our lunchtime to figure out how we can stop the hurt, and came up with Batman built using Serverless. Most of our infrastructure is managed in a single place (cloudformation) with a set of tags associated with each stack including the slice (or branch name), and the cloudformation stack keeps a created date & last update date. Utilizing this information Batman runs every night to cleanup stacks that are on non-master branches where the stack hasn't been updated for 7 days. This is usually enough time for us to have tested the branch and integrated it into the master stack.

We've also recently added a webhook trigger that deletes a stack when the branch is deleted in git which saves us up to 7 days of running costs.

We've stopped over 300 stacks in a month, and at a guesstimate of $1 per day running costs for a stack, the dollars quickly add up.

There are many ways to save costs, and we've found that for our journey into AWS, Batman has come about at the right time. Not only has it been interesting to explore serverless, our team got to do a lot of the work with some mob programming which was really fun (more to come).

Postgres search - overlapping arrays

Postgres Overlapping Arrays

We've got some spare food hanging around that we need to get rid of, but have to figure out which of our animals will eat it. Luckily we've recorded what each of them eat in our database! Given the three animals

{ 'species': 'cat', 'name': 'frank', 'eats': ['fish', 'biscuits', 'pudding'] }
{ 'species': 'dog', 'name': 'frankie', 'eats': ['fish', 'vegemite', 'cat'] }
{ 'species': 'snake', 'name': 'francine', 'eats': ['biscuits'] }

We want to find who will take the fish and pudding that we have sitting around. We'll get them out of their cage and bring them to the one spot to feed them. How can we figure this out from our data?

If the document was modelled to it's 3rd normal form in sql we could do some joins and figure it out without too much drama. To do this in Postgres with our jsonb storage a simple way to achieve the same outcome is to store the values we want to search on - eats in our case - is to store those values in their own column as an array (in addition to the jsonb column) and then use the Postgres array comparisons to do a search where the eats column contains any of ['fish', 'pudding']. What does this look like using Massive?

db.run(`select * from animals
  where eats && $1`,
  ['fish', 'pudding'],
  (err, results) => {
    // do something interesting with the results
})

Note that you'll need to add the extra column manually and populate it when saving a document:

db.animals.insert({body: jsonDocument, eats: jsonDocument.eats}, (err, results) => {})

Postgres - partial text search

Postgres Partial Text Search

Given the three documents below, we want to find the animals with name like 'frank'

{ 'species': 'cat', 'name': 'frank' }
{ 'species': 'dog', 'name': 'frankie' }
{ 'species': 'snake', 'name': 'francine' }

If this were sql we would search name like %frank% we'd expect to get the first two results. Postgres & Massive gives us a way to search for equality in a document so:

db.animals.findDoc({name: 'frank'}, (err, res) => { // do things with results })

Would only return the first result. (Note that Massive creates an index for your jsonb document if you let it create the table).

To search for partial results we need to add a Trigram Index - this breaks up text into three character chunks and gives a weighting to the search term match, rather than giving us an exact match or nothing. For the example above, we would break out the name property into another column (Massive creates a 'search' column which you could populate, or just make your own) - at this point we lose some of the ability to use Massive findDoc functions, but can still use the where function.

Creating the record would now look like

db.saveDoc("animals", {body: jsonDocument, name: jsonDocument.name}, function(err,res){});

And to query:

db.animals.where("name like $1", ['%frank%''], (err, result) => { // do something with the result });

Postgres - a document database?

Storing large json documents in SQL servers is painful. You first need to map the object out to a relational structure, then deal with putting it back together when you're querying. In addition you probably need to use some ORM to store your objects. All of which makes me sad.

With Postgres, there is a great feature that lets you store your json object into a column, and do queries on the json natively in the database!

I'm using Massive.js for working with Postgres in node, which has first class support for jsonb storage and makes the query syntax for working with jsonb documents a bit nicer. It also lets you write your own SQL when you need to do some more custom queries.

If you don't want any other columns in your table (it'll give you id and created date for free), you can use the saveDoc function (more on the doc storage syntax here):

db.saveDoc("my_documents", jsonDocument, function(err,res) {
    //the table my_documents was created on the fly with id and created date
    //res is the new document with an ID created for you
});

If you want to store some more data in your table then you can:

• Manually create the table yourself prior to using Massive
• Let Massive create the initial table for you, then add columns as needed (Massive will also create the primary key and auto-incrementing function which is handy)

We'll look at some querying strategies in the next post (there is documentation in the json section of the github repo which is a great place to start)

Testing SQL Server Connections

Testing SQL Server Connections
I found this gem today http://blogs.msdn.com/b/steverac/archive/2010/12/14/test-remote-sql-connectivity-easily.aspx trying to confirm if a windows server had access to a SQL instance (thinking I'd have to install something and do 10 restarts) - turns out you can create a .udl file which will bring up simple window to test your database connection!
Enjoy

One of the issues of infrequent releases.....

It's been a month since our last deployment, and we've got two features that's have been sitting in code for longer than that, and one that's been recently developed. One feature is 'hidden', the other isn't functional yet (a couple of external dependencies).

There are a couple of problems here:

• Not getting feedback on features that are ready to be released
• Keeping the information in your brain so that you can fix any bugs that arise when the feature finally gets released
• Trying to remember all the little bits that need to be completed to make the feature ready to release

A lot of this has to do with the flow of development, and perceived issues with releasing small features often. But the one that has kept on creeping up on me in the last couple of weeks is the realisation I want to keep adding new features, little ones that I think will make a difference, but aren't really necessary or proven to be required. 

This mentality comes from a "we're not going to be able to push anything new for another month", so try to pack everything in. It means that the code isn't as lean as it should be. Instead of waiting for feedback, reacting and pushing a new release, we just assume what people are after. 

Have you found this to be an issue when you're doing infrequent releases?

Raven DB Multi Map Index

I'm going to be looking at getting multiple document types returned from raven querying some properties that both objects have.

We're going to be using Animals and Vehicles, and we want to be able to figure out which animals and vehicles operate on land. These documents don't share inheritance, so we can't just search the base properties - what we need is a mapping that collects all the common properties of these documents.

The objects are:

We need to start by mapping the common fields to a new object for use in our LINQ query.
We also create a mapping for each object type that we want to index, selecting the common fields into an anonymous type.

You may have heard a lot of talk about Map/Reduce in the document database world, but for this particular query we're just interested in the Map part - we don't want to reduce the results to a common object but rather return the original objects.

Now that we've defined our Map we need to tell Raven about it which is really easy. Just after calling DocumentStore.Initialize() you call IndexCreation.CreateIndexes(typeof(AnimalsAndVehicles).Assembly, DocumentStore) which takes care of creating the index, and now it's ready to use!

To use the MultiMap: The key part of this is the As<dynamic>() - this will let us return any object from the query.

And the magic:

A couple of helpful links in putting this together and thinking about how the mapping works were the Raven Documentation and Daniel Lang's blog about Multi Map Indexes

Basic Raven DB Indexing and Querying with Lucene

We've got up and running with Raven DB for our new project, which has been a delight to use from a code perspective, but one of the big gaps for me is figuring out how to do SQL like queries. I'm not too bad at putting a SQL query together as something I do semi-regularly as part of my job, so I thought it shouldn't be too much of a stretch to start pulling some statistics.

I was wrong.

If you're using the .NET client, this is not something you particularly need to worry about as Raven will dynamically create indexes for you based on your LINQ query, which is awesome. My particular scenario is using the Raven web admin pages to do a query that I'm not particularly interested in coding - I just want to see some numbers quickly.

I'll talk you through the steps that I've gone through to get a simple query working, hopefully its of some benefit to you. To start with you should know that RavenDB search is powered by Lucene - we'll be looking at that in a bit. I'm going to start with a data set of animals :

What I'd like to see in the browser is a list of all the living land animals. What this means is we want to do a SQL equivalent of where on 2 of the fields: Lives AND Status.

To make these two columns queryable we need to make an index - unlike SQL we can't just query whatever we want whenever we want - we've got to tell Raven which fields we're going to do a query on. To do this jump over to the 'Indexes' section and 'Create an Index'. What we're going to do is create a document that maps to the columns we need to query, so give your index a name and jump into the 'Maps'.

A breakdown of the parts:

• from animals: animals is just the alias for the document we're going to refer to in the select
• in docs.SimpleIndexExample: this provides the source for the map - docs is generally all documents in the database, and SimpleIndexExample is the document type
• animals.Lives and animals.Status are the two columns we want to query on

Getting back to the point, I'm going to put together a query to get a list of all living land animals. Now that the index is created, we can click on it and use it to do a query (it'll look different depending on which version of Raven you've got running, but hopefully you'll be able to find the 'Query' section)

Creating the query is just a matter of telling Raven which field you want to query on, and value to restrict it to, so it should look something like this (by the way, you can get some auto-completion on the map field by pressing ctrl + space):

If you haven't picked it up, there's a problem with our result set - we're getting a Shark and TRex - this is not good! The problem is Lucene is looking for anything that matches the query, effectively doing an 'OR' on our statement. What we need to do is tell Lucene to require that a certain value is present - this is just putting a '+' symbol at the start of each required field:

For heaps more info about using Lucene to query have a look at the Lucene Query Parser Syntax Page

Please let me know if you have any comments, ways I can think about this better as this is still a very new area for me, or if you've found it helpful.

Task based workflow with NServiceBus and SignalR

Over the last six months our team has been working on breaking our system down into a whole lot of distinct, reusable components. Most of the work is completely automated, but there is always the case where we need some human input.

For each of these scenarios we've created very specific task based UI's, so we can taylor the screens to the exact need of the task. This has meant that we've been a lot more focused on solving a real business problem rather than just trying to jam as much information on the screen at once.

Up to this point, each task has been very distinct from the others, without too much consideration of the workflow that would really take place. There are a couple of options for how the tasks could be completed:

• Task centric users who complete just one type of task, and who don't particularly worry about the work flowing from their action
• Customer centric users who might need to call the customer to get some extra details, and are very interested in the way the rest of the work flows, as they want to be able to get all the information required to complete the whole transaction while on the phone. 

Our system currently supports task centric users, but we need a solution to customer centric solutions. For that, I had seen and heard a bit about SignalR:

Async signaling library for .NET to help build real-time, multi-user interactive web applications.

Along with listening to Hanselminutes, one of the posts that got me started was push notifications with nservicebus and signalr, which was right up my alley as we're using NServiceBus as our underlying infrastructure for messaging. The only difference in our project was using MVC3 for the web app, so I avoided the EventStream and created a class implementing SignalR's Hub - which was super easy.

I got a bit confused about how to get the Clients who are connected to my implementation of the Hub, but this stackoverflow answer from David Fowler helped me get the clients, so I could start notifying away!

Within a day I had a working notification system working! It definitely requires more refinement to make it more useful to our users, such as only notifying some people of the changes (trying to identify who 'owns' a transaction and wants to see further tasks) and some UI design to make things a bit friendlier, but it's definitely been really fund and easy to work with.

If you're working with task based UI's how do you create customer centric workflow?

Asynchronous Testing Woes

Over the last six months I've been having fun working with NServiceBus producing our new app at work. It's been great splitting our system up into functional bits (that we can hopefully reuse). Our current requirements mean we have one big saga (long running process) that uses a whole lot of handlers to get the job done.

This is awesome, as we've been able to attach a problem by talking about what the solution should be, splitting up into pairs and work on our own bit of the solution, then get back together and figure out our next bit of work, just having to define our messaging requirements collaboratively before we start.

We've also invested in writing specification tests with SpecFlow - taking a business scenario and making sure that our system can handle it from end to end. While this has been fun to learn, it's been hard work to get working due to the async nature of our system. What we've ended up doing is firing off the spec and then waiting 10 seconds until we check our auditing system to see if the saga has completed. This works fine for most of the tests, but there are often one or two specs that fail, but when you check the audits, everything has worked properly.

It would be awesome in this case to have a spec subscribe to the auditing events (currently there isn't an event published saying a saga has completed, but commands are sent to the auditor) - in that case the test could just wait until the publish, but it would also need a timeout (for when the saga actually fails).

This brings me to one problem that I've had with my specflow tests - the amount of time they take to run. These aren't like unit tests that we run all the time and only take a few seconds, the suit of tests takes minutes to run, so I'm looking to see if there are any solutions to run them in parallel. At the moment we use the NUnit library to implement the tests, but I have seen that MbUnit has a parallel library which looks interesting.

How do you do integration tests? Do you even do them?

Starting on the wrong foot

I've just joined a new project as a team member - my role is to write software. I'm keen to start off with domain driven design, really connecting with the users that know the processes and understand why things happen. At the moment we're writing some code to test some of our ideas to make sure we've got the skills we think we do.

I do think that domain driven design will be tough, especially getting buy in from managers to take the time of their domain experts to work through how they work and how they think, but that's all yet to come - our business analyst has done some great work on gathering initial requirements, so I think people are ready for the discussion that will come soon.

One of the big surprises for me is the lack of 'project team' or culture that has been created. Now, I work as a 'resource' (how good does that make you feel?) underneath my manager who has been more involved in the planning phase, but now that I'm actually part of the team, surely there should be some feeling of being part of a team? Bas de Baar of Project Shrink has quite a lot of articles about the culture in projects - well worth your time to have a look at. So I'm wondering what to do about this. It's got me quite frustrated over the last few weeks, especially as having tough conversations is made harder if you don't have a basis to work from.

My current plan is to organise a lunch outing, it'll at least be a start. I'd also be keen to go lawn bowling, but I'm not sure if anyone else would be keen, so it might not be so much of a winner for team culture.

Language difficulties

I went to the optometrist a few days ago. The optometrist that did my examination was really thorough and helpful, but there were a few things in the way she spoke that got my attention.

I'll talk about the points, and about how it can relate to working as a software developer.

Timing: when I'm supposed to choose from 2 lenses, seeing 3 lenses doesn't help. There were occasions when I didn't know which lense was 'one' or 'two'. Agreeing on which one I thought was better was also tough because I would say 'one' when specifying the better lense, and she would say 'one' - I wasn't sure if that was a question or statement, so wasn't sure if she'd noted my choice and had moved to the next test, or if she was checking what my decision was.

Suggestive language: The choice between two lenses shouldn't be influenced by anything other than what I'm seeing - several times the optometrist said 'one is better isn't it', putting into my mind that I shouldn't be choosing two.

Rewarding subjective choices: Similar to suggestive language, rewarding one choice or another makes you question if you made the 'right' choice.

Sitting in the chair being examined got me thinking about how I talk to people when I'm gathering requirements, talking to people about the problems they're trying to solve and discussing problems with colleagues.

I think its easy to jump to a solution, or assume requirements before allowing the subject matter expert really explain the situation. Just using neutral language to ask questions rather than a suggestive question, allowing them time to give more information.

When providing feedback its important to be really clear that you're providing feedback. Using phrases such as 'I'm going to summarise what I think we've talked about, if I've left anything out or got something wrong please tell me'.

This is just another great reason for software to be released often, it lets the discrepancy between my understanding of the situation and the users be exposed. Even when we work hard at understanding each other, it hits the road when they actually have to use what you've made.

In any case, I got some new contacts, and they're all sweet!

Raven DB & ASP.NET MVC routing

I've experimented a little with Raven DB, and have done some integration with ASP.NET MVC 3 (cause it's so easy to get things started).

The first problem I came up against was the default way that RavenDB stores the object keys :

profiles / 3074 (Profiles)

This presents a problem when you want to work with the MVC routing, which uses the '/' symbol in the routing - {controller}/{action}/{id}.

public ActionResult Edit(string id)

MVC interprets this as wanting to send the id (id e.g. 3074) to the action (object e.g. profiles). Instead, we want to pass the whole string (profiles/3074) as the id to the controller action (Edit).

To start, we need to change the type of id to string, as it defaults to int. Secondly, we need to change the format of the object key - there are other ways of attacking this problem, but I found changing the default separator from '/' to '-', this happens when you initialize the document store shown below

var documentStore = new DocumentStore { Url = "http://localhost:8080/" };

documentStore.Initialize();

documentStore.Conventions.IdentityPartsSeparator = "-";

We end up with documents in our store looking like this, and passing through to the correct controller!

profiles-6146 (Profiles)

Stand up

Danielle on my Desk
Originally uploaded by riebschlager

A couple of weeks ago I experimented with a standing desk, which meant dragging my monitors onto a shelf, putting my keyboard up higher and generally shuffling my desk about. It wasn't to be a permanent solution, but more to see if I thought it would be worth getting a permanent setup.

The first couple of days were pretty painful, it's been a while since I've stood up all day, but after a weekend I was feeling good. I felt like I had a lot more focus and didn't waste as much time on facebook and twitter, which was nice to feel that I was getting more done. I think it's also great for programmers, as pair programming would work well as opposed to having a corner sitting desk (which is what I currently work at), allowing better access to swap the keyboard and improved screen visibility.

Unfortunately my wrists weren't a big fan of the awkward position that I'd placed my keyboard in - without a solid desk to rest on they just started to flare up. So, I'm back sitting down waiting for my wrists to recover, and hopefully get my desk setup properly.

If you're interested in standing desks, Lifehacker is a great place to check out to start with.

Do you have a non-standard workplace? How did you make it happen? Does your workplace encourage you to work in a way that helps you work best, or do they want you to just sit there and stick to the standard way of doing things?

Web Testing

Have had a bit of fun doing some web testing over the last week. Mostly I work on internal systems, so don't get to have fun with user interfaces.

I've been using Selenium RC, with the aid of Selenium IDE in Firefox to figure out some XPath details.

A couple of things that I've learn't over the last couple of weeks:

• Using XPath to get elements in Internet Explorer is super slow.
• Selenium RC is great, but it's great to have the IDE just to test out roadblocks
• The .Click event mimicks a keypress on an element instead of a mouse click - I found some screen reader popups that only appear with keypress.
• You can't interact with Print / Save dialogs directly (although there are other tools for testing these components)

My tests are mostly checking that elements exist and have the appropriate classes / id's, so I'm yet to see how it works with positioning and styles being applied to the page. 

I'm also keen to use the css element locator - it's supposed to loads faster than finding elements with XPath.

If you're interested in Selenium, check out the proposed Stack Exchange site and give it an up vote. 

Where does your seat sit?

1-12-10 Another Day in the Cube
Originally uploaded by Patric Woessner

At work today, we had a bit of a chat about our physical work environment. Currently we've got an open plan setup happening, everyone faces into their corner of the workspace, and we've got space for a whiteboard. One of the issues with the current setup is that pair programming and code reviewing is a bit hard - trying to fit 2 people into a corner doesn't really work very well, and we want to start doing more code reviews to up the quality of our work.

I know that some people are big advocates of the 'everyone get's an office' idea, and have been reading Joel Spolsky's blog - particularly about office space. I think it'd be great to have my own office, but we don't really have the space, so we're trying to think about how we can work better in the space that we've got.

One idea is to have a large table that we all work at in the middle of the area, with whiteboards around the outside so we can draw and discuss ideas. Another possibility is to all work around the edge of the area, facing the wall, but not have anyone sitting in the corners - which will help with pairing and collaboration. A space would be reserved for a whiteboard, and we could wheel in another whiteboard in and out when we needed.

Anyway, hopefully over the next couple of months we can figure out a way to work better together, which may include a redesign of our physical work environment.

How do you work best with your team? Do you sit near each other? How does working remotely affect your collaboration?

Short sighted learning

Focus
Originally uploaded by Dani Ihtatho

I've been enjoying learning about a number of aspects of the .net world over the last year or so. Recently, I've been creating WCF Services - being new to the service creation world, I had to start from scratch, which is always a good place to start.

My problem, however, is that I wanted to learn enough just to get something up and running. I find that the problem with learning by google (or Stack Overflow, MSDN, blogs) is that there's no one pushing you to get the complete view of a topic - not that people don't offer the information up, it just seems irrelevant at the time when you're working so hard on the basics.

My lack of understanding was highlighted when doing some performance testing. I thought the service would create threads per call, but there are limits to the number of threads (which, of course, makes sense) - but what happens with those that don't get through to the service? Do the requests time out? What happens when the requests are being pased through from a DMZ? A standard request that worked normally as a one off test starts failing under load. None of these issues are problems within themselves, but a lack of understanding about how services generally work, how to queue up requests, how to handle concurrent requests all come crashing back to bite when you get an unexpected problem.

Talking about a different project (one that I wasn't involved in), I was saying to my colleague that I think it's not wise to check something in if you don't understand what it's doing - which I still think is right - but I didn't even know that I didn't know what I was doing!

To help my shortsightedness, I think I need to define some kind of learning plan that outlines all the parts of a new topic that I'm working on, even if it's just a quick read over a msdn article to see what 'bits' are talked about so I'm aware of their existance. I've even been considering buying books! The more I look at it, the more I think a good book is a great basis for learning a new topic - they're usually comprehensive in covering a topic, even if they don't touch on the detail of all aspects - google can come into help then.