Basic Raven DB Indexing and Querying with Lucene

We've got up and running with Raven DB for our new project, which has been a delight to use from a code perspective, but one of the big gaps for me is figuring out how to do SQL like queries. I'm not too bad at putting a SQL query together as something I do semi-regularly as part of my job, so I thought it shouldn't be too much of a stretch to start pulling some statistics.

I was wrong.

If you're using the .NET client, this is not something you particularly need to worry about as Raven will dynamically create indexes for you based on your LINQ query, which is awesome. My particular scenario is using the Raven web admin pages to do a query that I'm not particularly interested in coding - I just want to see some numbers quickly.

I'll talk you through the steps that I've gone through to get a simple query working, hopefully its of some benefit to you. To start with you should know that RavenDB search is powered by Lucene - we'll be looking at that in a bit. I'm going to start with a data set of animals :

What I'd like to see in the browser is a list of all the living land animals. What this means is we want to do a SQL equivalent of where on 2 of the fields: Lives AND Status.

To make these two columns queryable we need to make an index - unlike SQL we can't just query whatever we want whenever we want - we've got to tell Raven which fields we're going to do a query on. To do this jump over to the 'Indexes' section and 'Create an Index'. What we're going to do is create a document that maps to the columns we need to query, so give your index a name and jump into the 'Maps'.

A breakdown of the parts:

• from animals: animals is just the alias for the document we're going to refer to in the select
• in docs.SimpleIndexExample: this provides the source for the map - docs is generally all documents in the database, and SimpleIndexExample is the document type
• animals.Lives and animals.Status are the two columns we want to query on

Getting back to the point, I'm going to put together a query to get a list of all living land animals. Now that the index is created, we can click on it and use it to do a query (it'll look different depending on which version of Raven you've got running, but hopefully you'll be able to find the 'Query' section)

Creating the query is just a matter of telling Raven which field you want to query on, and value to restrict it to, so it should look something like this (by the way, you can get some auto-completion on the map field by pressing ctrl + space):

If you haven't picked it up, there's a problem with our result set - we're getting a Shark and TRex - this is not good! The problem is Lucene is looking for anything that matches the query, effectively doing an 'OR' on our statement. What we need to do is tell Lucene to require that a certain value is present - this is just putting a '+' symbol at the start of each required field:

For heaps more info about using Lucene to query have a look at the Lucene Query Parser Syntax Page

Please let me know if you have any comments, ways I can think about this better as this is still a very new area for me, or if you've found it helpful.

Task based workflow with NServiceBus and SignalR

Over the last six months our team has been working on breaking our system down into a whole lot of distinct, reusable components. Most of the work is completely automated, but there is always the case where we need some human input.

For each of these scenarios we've created very specific task based UI's, so we can taylor the screens to the exact need of the task. This has meant that we've been a lot more focused on solving a real business problem rather than just trying to jam as much information on the screen at once.

Up to this point, each task has been very distinct from the others, without too much consideration of the workflow that would really take place. There are a couple of options for how the tasks could be completed:

• Task centric users who complete just one type of task, and who don't particularly worry about the work flowing from their action
• Customer centric users who might need to call the customer to get some extra details, and are very interested in the way the rest of the work flows, as they want to be able to get all the information required to complete the whole transaction while on the phone. 

Our system currently supports task centric users, but we need a solution to customer centric solutions. For that, I had seen and heard a bit about SignalR:

Async signaling library for .NET to help build real-time, multi-user interactive web applications.

Along with listening to Hanselminutes, one of the posts that got me started was push notifications with nservicebus and signalr, which was right up my alley as we're using NServiceBus as our underlying infrastructure for messaging. The only difference in our project was using MVC3 for the web app, so I avoided the EventStream and created a class implementing SignalR's Hub - which was super easy.

I got a bit confused about how to get the Clients who are connected to my implementation of the Hub, but this stackoverflow answer from David Fowler helped me get the clients, so I could start notifying away!

Within a day I had a working notification system working! It definitely requires more refinement to make it more useful to our users, such as only notifying some people of the changes (trying to identify who 'owns' a transaction and wants to see further tasks) and some UI design to make things a bit friendlier, but it's definitely been really fund and easy to work with.

If you're working with task based UI's how do you create customer centric workflow?

Asynchronous Testing Woes

Over the last six months I've been having fun working with NServiceBus producing our new app at work. It's been great splitting our system up into functional bits (that we can hopefully reuse). Our current requirements mean we have one big saga (long running process) that uses a whole lot of handlers to get the job done.

This is awesome, as we've been able to attach a problem by talking about what the solution should be, splitting up into pairs and work on our own bit of the solution, then get back together and figure out our next bit of work, just having to define our messaging requirements collaboratively before we start.

We've also invested in writing specification tests with SpecFlow - taking a business scenario and making sure that our system can handle it from end to end. While this has been fun to learn, it's been hard work to get working due to the async nature of our system. What we've ended up doing is firing off the spec and then waiting 10 seconds until we check our auditing system to see if the saga has completed. This works fine for most of the tests, but there are often one or two specs that fail, but when you check the audits, everything has worked properly.

It would be awesome in this case to have a spec subscribe to the auditing events (currently there isn't an event published saying a saga has completed, but commands are sent to the auditor) - in that case the test could just wait until the publish, but it would also need a timeout (for when the saga actually fails).

This brings me to one problem that I've had with my specflow tests - the amount of time they take to run. These aren't like unit tests that we run all the time and only take a few seconds, the suit of tests takes minutes to run, so I'm looking to see if there are any solutions to run them in parallel. At the moment we use the NUnit library to implement the tests, but I have seen that MbUnit has a parallel library which looks interesting.

How do you do integration tests? Do you even do them?

Starting on the wrong foot

I've just joined a new project as a team member - my role is to write software. I'm keen to start off with domain driven design, really connecting with the users that know the processes and understand why things happen. At the moment we're writing some code to test some of our ideas to make sure we've got the skills we think we do.

I do think that domain driven design will be tough, especially getting buy in from managers to take the time of their domain experts to work through how they work and how they think, but that's all yet to come - our business analyst has done some great work on gathering initial requirements, so I think people are ready for the discussion that will come soon.

One of the big surprises for me is the lack of 'project team' or culture that has been created. Now, I work as a 'resource' (how good does that make you feel?) underneath my manager who has been more involved in the planning phase, but now that I'm actually part of the team, surely there should be some feeling of being part of a team? Bas de Baar of Project Shrink has quite a lot of articles about the culture in projects - well worth your time to have a look at. So I'm wondering what to do about this. It's got me quite frustrated over the last few weeks, especially as having tough conversations is made harder if you don't have a basis to work from.

My current plan is to organise a lunch outing, it'll at least be a start. I'd also be keen to go lawn bowling, but I'm not sure if anyone else would be keen, so it might not be so much of a winner for team culture.

Language difficulties

I went to the optometrist a few days ago. The optometrist that did my examination was really thorough and helpful, but there were a few things in the way she spoke that got my attention.

I'll talk about the points, and about how it can relate to working as a software developer.

Timing: when I'm supposed to choose from 2 lenses, seeing 3 lenses doesn't help. There were occasions when I didn't know which lense was 'one' or 'two'. Agreeing on which one I thought was better was also tough because I would say 'one' when specifying the better lense, and she would say 'one' - I wasn't sure if that was a question or statement, so wasn't sure if she'd noted my choice and had moved to the next test, or if she was checking what my decision was.

Suggestive language: The choice between two lenses shouldn't be influenced by anything other than what I'm seeing - several times the optometrist said 'one is better isn't it', putting into my mind that I shouldn't be choosing two.

Rewarding subjective choices: Similar to suggestive language, rewarding one choice or another makes you question if you made the 'right' choice.

Sitting in the chair being examined got me thinking about how I talk to people when I'm gathering requirements, talking to people about the problems they're trying to solve and discussing problems with colleagues.

I think its easy to jump to a solution, or assume requirements before allowing the subject matter expert really explain the situation. Just using neutral language to ask questions rather than a suggestive question, allowing them time to give more information.

When providing feedback its important to be really clear that you're providing feedback. Using phrases such as 'I'm going to summarise what I think we've talked about, if I've left anything out or got something wrong please tell me'.

This is just another great reason for software to be released often, it lets the discrepancy between my understanding of the situation and the users be exposed. Even when we work hard at understanding each other, it hits the road when they actually have to use what you've made.

In any case, I got some new contacts, and they're all sweet!

Raven DB & ASP.NET MVC routing

I've experimented a little with Raven DB, and have done some integration with ASP.NET MVC 3 (cause it's so easy to get things started).

The first problem I came up against was the default way that RavenDB stores the object keys :

profiles / 3074 (Profiles)

This presents a problem when you want to work with the MVC routing, which uses the '/' symbol in the routing - {controller}/{action}/{id}.

public ActionResult Edit(string id)

MVC interprets this as wanting to send the id (id e.g. 3074) to the action (object e.g. profiles). Instead, we want to pass the whole string (profiles/3074) as the id to the controller action (Edit).

To start, we need to change the type of id to string, as it defaults to int. Secondly, we need to change the format of the object key - there are other ways of attacking this problem, but I found changing the default separator from '/' to '-', this happens when you initialize the document store shown below

var documentStore = new DocumentStore { Url = "http://localhost:8080/" };

documentStore.Initialize();

documentStore.Conventions.IdentityPartsSeparator = "-";

We end up with documents in our store looking like this, and passing through to the correct controller!

profiles-6146 (Profiles)

Stand up

Danielle on my Desk
Originally uploaded by riebschlager

A couple of weeks ago I experimented with a standing desk, which meant dragging my monitors onto a shelf, putting my keyboard up higher and generally shuffling my desk about. It wasn't to be a permanent solution, but more to see if I thought it would be worth getting a permanent setup.

The first couple of days were pretty painful, it's been a while since I've stood up all day, but after a weekend I was feeling good. I felt like I had a lot more focus and didn't waste as much time on facebook and twitter, which was nice to feel that I was getting more done. I think it's also great for programmers, as pair programming would work well as opposed to having a corner sitting desk (which is what I currently work at), allowing better access to swap the keyboard and improved screen visibility.

Unfortunately my wrists weren't a big fan of the awkward position that I'd placed my keyboard in - without a solid desk to rest on they just started to flare up. So, I'm back sitting down waiting for my wrists to recover, and hopefully get my desk setup properly.

If you're interested in standing desks, Lifehacker is a great place to check out to start with.

Do you have a non-standard workplace? How did you make it happen? Does your workplace encourage you to work in a way that helps you work best, or do they want you to just sit there and stick to the standard way of doing things?