Update: Exciting times! Lots of things have changed in the fortnight since I wrote this post, and I’m no longer going to work at InfoMentor. Instead I’m proud to say I’ve founded a new software house down here in Devon, 36degrees, alongside my former employers at Tech 13. Really excited, it’s a great opportunity and I really hope I can make a good go of it! :)


I recently moved from Birmingham down to Devon, and began working from home. Unfortunately, (at this point in my life at least) the isolation and lack of social interaction that comes from working from home was not for me, and so it is with a very heavy heart than I am leaving my current job at Tech13 to work for InfoMentor developing collaborative online learning tools. Don’t get me wrong, I’m very excited for the new experience at a new company, but it’s always difficult to leave, especially a job that I enjoyed so much.

I’ve been with Tech13 for a little over a year, and am incredibly grateful for the opportunities and experiences I have had whilst working there. It’s a great team, and I’m particularly thankful to have been mentored by Paul Cox, an incredibly talented developer whose passion for learning has been hugely infectious and has fundamentally shaped the way I approach the betterment of my craft. Through Paul I learnt the joy of immersing myself into a community that I really had a limited knowledge of beforehand. I feel shocked now when I meet a developer that doesn’t read blogs or listen to podcasts, and who is unfamiliar with the great technologies and open-source projects that exist outside of his/her own little bubble; but that was once me. It was Paul who reccommended I start this blog, Paul who advised me to really dig deep into the systems that I’m using, and Paul who showed me that in order to really succeed you could not be afraid to take a few chances.

I wrote a blog post at the start of the year on my experiences so far in the real world of software development, and on the decision to pursue experience over further education. I’m only a little further on now, so I’m probably still not qualified to answer my own question: “Did I make the right decision to work rather than to continue down the educational track?” That said, I stand by it being one of the best decisions that I ever made.

Obviously it’s tough to be moving on, but I’m very excited for the next chapter of my career to begin and for the new experiences and opportunities that working for InfoMentor will bring.

Oh yeah, and Paul, in case you happen to be reading this: for god’s sake, start your bloody blog! I’m going to miss hearing the things you have to talk about, so please write them down so I can still get my fix!

<div class='box'>
    <h2 style='-webkit-user-select: none;'>Title</h2>
    <div class='boxSlide'>
        <div class='boxBg'>
            <div class='boxContent'>
                content...
            </div>
        </div>
        <div class='boxFooter'><img src='img/ws_box_footer.png'></div>
    </div>
</div>

Apologies for the above code. I realise it’s not very pleasant to look at, but unfortunately I was working on a website not too long ago which was based off of an online template, and this was the structure I’d been asked to use to add some fancy, styled boxes to a whole bunch of pages so as not to break the existing site’s css.

Now that’s quite a lot of markup for a simple section – please understand I’d be far happier using cleaner and more semantic markup, such as section, article, or even a single div if needsbe – but unfortunately that wasn’t an option. However, even if that was the code that I had to use, it sure as hell wasn’t the code I was going to write. Not over and over and over…

I first wrote a HtmlHelper extension method which would allow me to use the syntax @Html.Section(“header”, “content…”), which was definitely a marked improvement on all that code, and lead to much more readable source.

public static MvcHtmlString Section(this HtmlHelper html, string header, string content) {
   return MvcHtmlString.Create(
              "<div class='box'>" +
                  "<h2 style='-webkit-user-select: none;'>" + title + "</h2>" +
                  "<div class='boxSlide'>" +
                      "<div class='boxBg'>" +
                          "<div class='boxContent'>" +
                              content +
                          "</div>" +
                      "</div>" +
                      "<div class='boxFooter'><img src='img/ws_box_footer.png'></div>" +
                  "</div>" +
              "</div>";
}

Job done. Well, kind of. This worked well for just text-only sections. A section containing multiple paragraphs, images, or other html elements was possible, but looked a mess and was harder to read that the original syntax! It completely ruled out the option of using more complicated contents in a section – ie. if I wanted a section to contain a form. Furthermore, these boxes were nestable – so one box could potentially contain multiple. Since they took an string as content and returned an MvcHtmlString, then in addition to these problems I would have had to keep adding .ToString() to the end of each nested Html.Section().

@using (Html.BeginForm()) { ... }

How many times have we all used the above code? And what does it do? Well that’s easy, it’s an HtmlHelper for creating forms. The interesting thing with BeginForm, however, is that unlike most other HtmlHelpers, this allows for start and end tags, but with any other content nested within the curly braces (or if you prefer, open/close mustaches).

Oh what a coincidence, that sounds almost exactly like what I need to do!

So how does it work? Well fortunately, the solution is brilliantly easy. When you call BeginForm, it first outputs the opening

tag, not by returning an MvcHtmlString, but by writing directly to the view context using HtmlHelper’s htmlHelper.ViewContext.Writer.Write(string). It then returns a class implementing IDisposable (in this instance, an MvcForm), which is passed the view context when instantiated. All this class does is to write out to the view context when it is being disposed of – which as we all know is at the close of the braces.

Therefore, the code needed to allow me the freedom to write my sections as

@using (Html.Section("title here")) {
    ... any content I want! ...
}

is as follows:

public static HtmlSection Section(this HtmlHelper html, string title)
{
    html.ViewContext.Writer.Write(
              "<div class='box'>" +
                  "<h2 style='-webkit-user-select: none;'>" + title + "</h2>" +
                  "<div class='boxSlide'>" +
                      "<div class='boxBg'>" +
                          "<div class='boxContent'>"
        );
    return new HtmlSection(html.ViewContext);
}


public class HtmlSection : IDisposable
{
    private readonly ViewContext _viewContext;

    public HtmlSection(ViewContext viewContext)
    {
        _viewContext = viewContext;
    }

    public void Dispose()
    {
        _viewContext.Writer.Write(
                          "</div>" +
                      "</div>" +
                      "<div class='boxFooter'><img src='img/ws_box_footer.png'></div>" +
                  "</div>" +
              "</div>"
        );
    }
}

Not long ago I blogged about auditing with NHibernate.Envers. I’m using it again in another project but I don’t want to audit all the fields on the entity; furthermore, Envers automatically tries to audit referenced tables, so tables that are not configured to be audited cause an error message: An audited relation from SomeAuditedEntity to a not audited entity NonAuditedEntity! Such mapping is possible, but has to be explicitly defined using [Audited(TargetAuditMode = RelationTargetAuditMode.NotAudited)]. Understandably I expected that adding this attribute to the properties that I didn’t want auditing would prevent this behaviour, but it didn’t. I then added it to the class that I didn’t want auditing, but the message still didn’t go away. A quick google revealed another similar attribute, NotAudited, but this didn’t work either.

Finally after hacking around with my code I stumbled across the solution; the properties to be excluded must be defined in the Envers configuration. Although I think I’d prefer to annotate the fields in the entity class itself, this is the solution that worked for me:

var enversConf = new NHibernate.Envers.Configuration.Fluent.FluentConfiguration();
enversConf.SetRevisionEntity<REVINFO>(e => e.Id, e => e.RevisionDate, new AuditUsernameRevInfoListener());
enversConf.Audit<SomeAuditedEntity>()
          .Exclude(x => x.NonAuditedProperty1)
          .Exclude(x => x.NonAuditedProperty2)
          .ExcludeRelationData(x => x.NonAuditedCollection;
configuration.IntegrateWithEnvers(enversConf);

I use two different types of auditing in my various projects – low-level auditing, where individual property changes are recorded, and high-level, where I record actions (ie. created x, edited y). For the high-level auditing, I record each ‘action’ using a class, which inherits from a base Audit class. The basic audit class is as follows:

public class Audit : Entity {
    public virtual string CreatedBy { get; set; }
    public virtual DateTime CreatedOn { get; set; }
    public virtual Func<HtmlHelper,MvcHtmlString> Text {
        get { return html > MvcHtmlString.Empty; }
    }
}

CreatedBy and CreatedOn are both fairly obvious properties, recording when the audit was created and by whom. The Text property is used for showing the audit information in views and provides an easy and highly flexible way to display the audit text, as will be demonstrated shortly. As I said, each auditable ‘action’ is represented using a new class. These classes are subclasses of Audit, and can contain additional fields. For example, I may have an audit for the creation of a new user:

public class UserCreateAudit : Audit {
    public virtual int UserId { get;set; }
    public virtual string Username { get; set; }
    public virtual Func<HtmlHelper,MvcHtmlString> Text {
        get {
            return html => MvcHtmlString.Create(
                string.Format("Created new user {0}", Username)
            );
        }
    }
}

In order to store the audit data, we can use NHibernate’s inheritance mapping. NHibernate supports three strategies for polymorphism – table-per-hierarchy, table-per-class, and table-per-concrete-class. Table-per-class and table-per-concrete class are similar, storing the data of different subclasses in separate tables, while table-per-hierarchy stores the Audit class and any subclasses in a single table, containing all the entities in all the classes. I use table-per-hierarchy because it uses fewer joins so reading/writing is faster, although the trade off is that there will be many irrelevant columns stored for each audit type. Modern database management systems work very efficiently, minimising the effects of storing null/redundant data.

Configuring NHibernate for Polymorphism

Setting up inheritance is really easy:

public class AuditAutoMappingOverride : IAutoMappingOverride
{
    public void Override(AutoMapping mapping)
    {
        mapping.DiscriminateSubClassesOnColumn("Type");
    }
}

Ok, that’s a bit unfair. We use FluentNHibernate.AutoMapper, so if you do too then you can simply override the mapping; for everyone else, sorry but you’ll have to look it up yourself! Honestly though, it isn’t hard! I really just wanted to show you how NHibernate acheives polymorphism: it uses an extra column, in our case called ‘Type’, which acts as a discriminator. Each subclass of Audit will have it’s own discriminator, which is by default the full name of the type, and this is how NHibernate can differentiate between audit types.

Why is this good for auditing then?

There are three reasons why using polymorphism and NHibernate’s inheritance mapping strategies work well with the style of auditing that I use:

  1. Using a separate class per audit type makes it really easy to maintain the various audit types and the data relevant to each one. For instance when auditing that a user created a new task, in addition to the CreatedOn and CreatedBy fields, I want to store the task id, the type of task created, and the due date of the task; which results in a clear to read audit subclass:
    public class TaskCreateAudit : Audit {
        public virtual int TaskId { get; set; }
        public virtual TaskType TaskType { get; set; }
        public virtual DateTime DueDate { get; set; }
    }

    Also, if multiple subclasses have the same property NHibernate will aggregate them all when mapping the database table. For example if I have another audit class that has the properties DueDate and AnotherProperty, then the resulting table will have the fields Type, CreatedBy, CreatedOn, TaskId, TaskType, DueDate and AnotherProperty (note only a single instance of DueDate)

  2. The text shown when displaying the audit data is highly customizable, as a result of using separate classes for each audit type. I find using a Func<HtmlHelper,MvcHtmlString>  is a really simple, elegant, and flexible way to determine how the audit information is displayed in the view.
    public class TaskCreateAudit : Audit {
        public virtual int TaskId { get; set; }
        public virtual TaskType TaskType { get; set; }
        public virtual DateTime DueDate { get; set; }
        public override Func<HtmlHelper,MvcHtmlString> Text {
            get { return html => MvcHtmlString.Create(
                "Created new task (id #" + TaskId + ")"
            ); }
        }
    }

    Using @Model.Text.Invoke(Html) in a view will display: Created new task (id #1001). If later on I decide that actually I would rather link to the task instead, and show the due date in the audit, all I need to do is change the class definition:

    public override Func<HtmlHelper,MvcHtmlString> Text {
        get { return html => html.ActionLink(
            "Created new task due " + DueDate.ToString(),
            "Index", "Tasks", new { id = TaskId }, null
        ); }
    }

    And now the audits will display as: Created new task due 01/01/2012

  3. Finally, aside from ease of use and flexibility, auditing in this way allows for really easy querying of the data. For instance, to query all audit entries, I can use:
    _session.Query<Audit>();

    Similarly, if I want to query only a certain type of audit, I can use:

    _session.Query<TaskCreateAudit>();

In most of the projects I’ve worked on, auditing has been a fairly high level affair – typically recording user actions, such as user edited entity x or user deleted child entity y. This has been adequate for most of our systems where we do not need to be able to see exactly all the modifications made to an entity. However, for a recent system this style of auditing has been causing issues; on several occasions we’ve had requests from clients saying “a property on this entity is not as expected, but it was correct x days ago, can you see who changed it?”. And unfortunately, unless only a single user has edited the entity, we cannot see who made the change. And that really isn’t good enough…

So I took a look at NHibernate.Envers, a project which facilitates easy entity versioning with NHibernate and would therefore enable me to save an in depth history of every version of a particular entity. The docs weren’t great I found, but Giorgetti Alessandro over at PrimordialCode has a great series of posts covering virtually everything you need to know: a quick introduction, querying (part 1 and part 2) and customising the revision entity. If you’ve got some time to spare, I’d suggest you to read the introduction before continuing, as it’s a great article with some really valuable information, and clearly demonstrates the database structures generated by Envers.

 Wiring Up .Envers with Existing Auditing

After reading the PrimordialCode posts, I was able to really quickly get Envers up and running. In order to link our existing high-level action based auditing up with the detailed low-level information provided by Envers, I needed to store the revision id as part of the existing audit, which was really easy to access using IAuditReader.GetCurrentRevision(bool persist)

[HttpPost]
public ActionResult Edit(int id, UserViewModel model)
{
    var auditer = _session.Auditer();
    var user = _userBuilder.BuildEntity(model, id);
    var revisionInfo = auditer.GetCurrentRevision<REVINFO>(true);

    /* Create an instance of UserEditAudit which references the
       revision info                                             */
    _session.Save(UserEditAudit.Create(
                        LoggedInUser.UserName,
                        user, revisionInfo));

    return RedirectToAction("Overview", new { id });
}

Displaying Audit Data

Now we have the revision id for the audit, we can create a view to display the revision data. Out of the box, Envers enables you to query entities at a particular revision, and also query the revision data (to get the timestamp). Unfortunately, using the built in audit reader generates very inefficient sql with multiple and unnecessary queries.

Furthermore, I wanted to be able to show not simpy the version of the entity, but the differences between it and it’s previous version. Envers doesn’t support this behaviour out of the box, so to implement it I needed to load the two versions of the entity and diff them myself. Because of the inefficiencies of looking up entity versions using Enver’s audit reader, I wrote a generic method that could loaded the two versions of any auditable entity using a single sql query with dapper:

private IEnumerable<T> EntityAndPreviousRevision<T>(int entityId, int revisionId) where T : Entity<T>
{
    var sql = string.Format(
        @"select * from dbo.{0}_AUD where {0}Id = @entityId and REV in (
         select top 2 a.REV
         from dbo.{0}_AUD a with (nolock)
         inner join dbo.REVINFO r with (nolock) on a.REV = r.REVINFOId
         where a.{0}Id = @entityId and a.REV <= @revisionId
         order by r.RevisionDate desc )", typeof(T).Name);
    return _connectionProvider
                    .Connection
                    .Query<T>(sql, new { entityId, revisionId });
}

I then diffed the two classes by simply using reflection to iterate through each of the properties on the entity and checking for any non-matches:

public class PropertyChange
{
    public string From { get; set; }
    public string To { get; set; }

    public bool IsDifferent()
    {
        if (From == null && To == null) return false;
        if (From == null || To == null) return true;
        return !From.Equals(To);
    }
}

private static Dictionary<string,PropertyChange> DifferencesBetweenRevisions<T>(T from, T to)
{
    var properties = typeof (T).GetProperties();
    var differences = new Dictionary<string, PropertyChange>();
    foreach (var property in properties)
    {
        var pc = new PropertyChange
        {
            From = (property.GetValue(from, null) ?? String.Empty).ToString(),
            To = (property.GetValue(to, null) ?? String.Empty).ToString()
        };
        if (pc.IsDifferent()) differences.Add(property.Name, pc);
    }
    return differences;
} 

Although cloud-based file storage has existed in some form for a long time, it is now starting to play a more prominent role in day-to-day use. Firstly, most servies are offering generous storage amounts for free accounts: DropBox offers 2GB, Google Drive offers 5GB, and SkyDrive offers 7GB. Upgrading the amount of available storage you have is cheap too, between $0.25 and a couple of dollars per extra GB There are also services such as Bitcasa (currently only in beta) which offer unlimited storage, which is pretty tough to resist.

Price aside, most importantly of all they make it really easy to share files between all the different platforms you own – your desktop, laptop, tablet, phone – and even enable you to have access to all of your files from another persons machine if needsbe.

Gone are the days where you would be forced to log onto a website, and upload your files a handful at a time through a web form (or worse, though some java multi-uploader applet). Typcially, all you need to do is install the app, which drops a virtual folder onto your computer. You can then proceed to use this folder like any other, integrating natively with your machine, meaning it’s super simple to use. Even my technophobe fiancé has a DropBox account, which she loves!

The final convenience that cloud-based file storage offers is easy integration of files with cloud-based apps. For instance, I can be working on a word document on my desktop at home, save it to my Google Drive file, and then jump on a train and continue working on it on my galaxy tab from Google Docs, and there are a number of similar file sharing scenarios available for most of the storage services available today.

My only concern with these virtual storage systems are with the security of the files. While I trust that these services have strong security in place, I’d still feel somewhat uneasy about using storing sensitive data on them. My other concern, though I’m sure completely unfounded, is with the longevity – I’d hate to store the only copy of an important document on a cloud service and risk it vanishing. For those reasons, I find myself using virtual storage systems mainly for backups, though I hope expect when I’m more comfortable with using these services my behaviour will shift to reflect that.

MongoDB is an open source database that is very scalable and high performing, and is fast becoming one of the more popular NoSql implementations. It uses a data format known as BSON, which is syntactically similar JSON but is saved in binary form for speed; is schema free, meaning that documents belonging to the same store need not necessarily have the same structure; and can store dates, object ids, binary data, regular expressions and code as well as the basic JSON types of string, integer, boolean, double, null, array and object.

Installing Mongo is really simple:

  • create a mongo file (not necessary but tidy) [mkdir /mongo]
  • pull mongo [curl http://downloads.mongodb.org/linux/mongodb-linux-x86_64-{{VERSION}}.tgz > mongo.tgz]
  • extract [tar xzf mongo.tgz]
  • create data file [mkdir -p /data/db/]
  • install [chown id -u /data/db]

Next install mongodb-php:

  • install dev tools (if necessary) [yum groupinstall "Developer Tools"]
  • check whether pecl needs upgrading (it did for me) and check mongo package available [pecl search mongo]
  • install [pecl install mongo] (if any problems, try creating mongo directory in /tmp)
  • restart apache [apachectl graceful]

If you’re anything like me, you’ll prefer a half-decent administration ui over the command line. I originally looked at phpMoAdmin, but for whatever reason it would die, giving me an HTTP 500 error when it did so but with no further guidance, every other time I accessed it (including navigating from page to page…). Even when it did work it looked awful and didn’t seem very friendly, so I gave up on that and installed RockMongo.

RockMongo was super simple to install – just a matter of cloning the respository onto the root of my server – and tweak the config if necessary (for me the defaults worked fine). The only problem that I had was that I couldn’t work out how to change the default username/password from admin/admin. It’s really easy though, was probably more to do with me being tired than it being complicated: simply change the line

$MONGO["servers"][$i]["control_users"]["admin"] = "admin";

to

$MONGO["servers"][$i]["control_users"][{{username}}] = {{password}};

in the config.php

I recently needed to implement a facility on a web page to download a file that was defined by a reasonably number parameters, stored on the page as JSON. The first thing I tried was to ajax the request, but that failed as the file never downloaded – I retrieved the data, but it essentially just got ‘swallowed’.

Next I tried using JSON.stringify to populate a hidden field with the JSON value and then submit that. The action that I was posting to was defined with the parameters separately, rather than just string, and I wasn’t able to edit it: I had hoped that ASP.NET MVC 3 might have been able to decode the string but it couldn’t, so that approach failed too. (If I’d been able to amend the action then I could have obviously deserialized the string, but I wasn’t allowed [boo!]).

So in the end I wrote a quick method that would populate my form with hidden fields representing the properties of my JSON model, and that ended up working nicely. Really simple, and would obviously have to be tweaked to handle nested JSON, but it got the job done:

function downloadCurrentReport() {
    var newHiddenInputs = "", i;
    for (i = 0; i < model.length; i++) {
        var nameBase = "model[" + i + "].";
        for (var prop in model[i]) {
            newHiddenInputs +=
                    hiddenField(nameBase + prop, model[i][prop]);
        }
    }
    $("#downloadForm").html(newHiddenInputs);
    $("#downloadForm").submit();
}
    
function hiddenField(name, value) {
    return "<input type='hidden' name='" + name + "' value='" + value + "'/>";
}

I’ve been reading recently about the new HTML5 specs, and one thing that really interested me was the Web SQL Database spec that would allow javascript developers to access a client-side database from within the browser, in order to save and manipulate data locally on the users machine. This would enable interactive javascript web applications to run offline, in a similar manner to how Google Gears works (think storing emails for access at a time when no internet connection is available). The WebDB API defined a relational database that could be queried using SQL. Unfortunately, as of 18th November 2010, this spec was canned, because it suffered from one fatal flaw: it used SQL. The problem was that in order to define a cross-browser compatible API, all vendors would need to implement the same database (or more specifically, the same form of SQL). The spec pushed SQLLite as the database implementation, which both Chrome and Safari agreed with, but since Microsoft wanted to use a version of SQL Server in IE, work on the specification was ceased.

The replacement for the WebDB API was the Indexed Database API. Unlike webDB, this was a NoSQL database implementation, which used object stores rather than the typical relational database implementation. The main problem with the IndexedDB API, it seemed to me, however, was that the syntax was just awful compared to it’s SQL alternative. The code samples found at Mozilla Hacks show this pretty well (although it seems that the point of the post is supposed to sing the advantages of IndexedDB over WebDB!). This example, taken from that post, shows the different code samples required to load and display all the kids in a database that have bought candy:

WebDB

var db = window.openDatabase("CandyDB", "1",
                             "My candy store database",
                             1024);
db.readTransaction(function(tx) {
  tx.executeSql("SELECT name, COUNT(candySales.kidId) " +
                "FROM kids " +
                "LEFT JOIN candySales " +
                "ON kids.id = candySales.kidId " +
                "GROUP BY kids.id;",
                function(tx, results) {
    var display = document.getElementById("purchaseList");
    var rows = results.rows;
    for (var index = 0; index &lt; rows.length; index++) {
      var item = rows.item(index);
      display.textContent += ", " + item.name + "bought " +
                             item.count + "pieces";
    }
  });
});

IndexedDB

candyEaters = [];
function displayCandyEaters(event) {
  var display = document.getElementById("purchaseList");
  for (var i in candyEaters) {
    display.textContent += ", " + candyEaters[i].name + "bought " +
                           candyEaters[i].count + "pieces";
  }
};

var request = window.indexedDB.open("CandyDB",
                                    "My candy store database");
request.onsuccess = function(event) {
  var db = event.result;
  var transaction = db.transaction(["kids", "candySales"]);
  transaction.oncomplete = displayCandyEaters;

  var kidCursor;
  var saleCursor;
  var salesLoaded = false;
  var count;

  var kidsStore = transaction.objectStore("kids");
  kidsStore.openCursor().onsuccess = function(event) {
    kidCursor = event.result;
    count = 0;
    attemptWalk();
  }
  var salesStore = transaction.objectStore("candySales");
  var kidIndex = salesStore.index("kidId");
  kidIndex.openObjectCursor().onsuccess = function(event) {
    saleCursor = event.result;
    salesLoaded = true;
    attemptWalk();
  }
  function attemptWalk() {
    if (!kidCursor || !salesLoaded)
      return;

    if (saleCursor &amp;&amp; kidCursor.value.id == saleCursor.kidId) {
      count++;
      saleCursor.continue();
    }
    else {
      candyEaters.push({ name: kidCursor.value.name, count: count });
      kidCursor.continue();
    }
  }
}

Pretty monstrous right? Which is a real shame, since IndexedDB has the potential to be a really, really useful in a client side developers toolkit.