I’ve been working recently on a new project, Jello, which I intend to be an easy to use expression/rule evaluator, and has just hit a useable-though-lacking-in-features state.

Why use?

I wanted Jello to be really simple to use, and think I achieved that. Before I began writing Jello I tried a number of different approaches, including using embeddable languages such as Boo or IronPython, parser generators such as Irony, and expression evaluators such as NCalc. I found that for my needs, to evaluate simple expressions, the embeddable languages were just overkill; parser generators worked well, but I had to write a lot of boilerplate each time to specify the core of the language; and the expression evaluators didn’t offer the flexibility I wanted; and so decided to roll my own. The result is a compact, simple and easy to use and (will be) highly configurable.

Typical use cases could be:

  • Decision/underwriting rules (I’ve used this in a document generator to allow users to specify criteria to determine if a part of a document is included or not. I’ve also implemented similar behaviour in the past (before writing Jello) for scripting rules to determine whether a buyer accepts a particular lead or not)
  • Evaluating simple expressions, such as calculating the max value for a field based on some criteria
  • Essentially, any time that you need your users to be able to have a high degree of flexibility when specifying rules/calculations, Jello is your friend

Syntax

Currently it has a very simple syntax consisting of basic equality/logical operators (&&, || and ==) and arithmetic (+, -, * and /), and supports basic data types: booleans, strings, dates and numbers (only decimal types to keep it simple – after all, Jello is intended for the users of your system). Date formats are not fixed allowing you to specify the format that best suits your needs – the default is to use the StandardDateParser which ships with Jello and uses the .net DateTime.TryParse(dateString, out date) method internally, but you can create a class which inherits from the IDateParser interface and parse dates however you’d like. Using the StandardDateParser you can also specify an IFormatProvider and DateTimeStyles if you’d like.

You can also use variables which take their values from a data source that is passed into the parse result when executing. Data sources are classes that inherit from a simple interface, IDataSource, which specifies only a single method: bool TryGet(string key, out object value). The only datasource currently shipped with Jello is a CompositeDataSource, which simply allows you to aggregate multiple data sources, though I intend to add a basic dictionary datasource shortly so that you won’t always have to create your own classes.

Modifiability

As I said before, the syntax is currently fixed, but my hope for Jello is that it will become flexible enough that it can be fit to most uses. The date parser is the first example of this to be added, since there are so many nuances with dates that I didn’t want to force users to use one specific date format, but I plan to add configurable operator overloading, functions and macros so that Jello can be bent to any will.

Todo:

  • Add operator overloading / functions
  • Support user-defined datatypes
  • Add macros to allow substituting operators (ie. using ‘and’ instead of ‘&&’)
  • Extend grammar to make it more than just expression evaluation; allowing for specifying/amending variables, if/else statements, loops and return statements.

If ever you accept user-written HTML code in your web applications, such as may be generated in a rich-text ‘wysiwyg’ text editor, it is vital that before displaying it back anywhere you first sanitize it. Sanitization is the process of removing potentially malicious code, primarily to prevent xss (cross-site scripting) attacks; and is generally achieved by allowing only a subset of tags and attributes in the submitted code and removing or encoding the rest.

I recently needed to do this, and a quick google turned up a project, patapage, which does just that. Although it is a java solution, there is a C# port written by Beyers Cronje; unfortunately it’s some seriously ugly code, being more-or-less a straight rip of the java version, just fixed to be valid c# code.

I realise that some people don’t think that is necessarily a bad thing, but I can’t stand to see eyesores, such as lowercase methods and type names instead of ‘var’, and so had to clean it up. I take no credit for any of the code, all I did was capitalize property/method names; replace some if/elses with ternary operators for terseness (where appropriate); replace type names with ‘var’; and change some arrays to IEnumerables (I hope this should give a bit of a performance gain, but I didn’t bother to check so don’t quote me on that).

HtmlSanitizer.cs

I’m currently working on implementing an issue tracker within a customer’s web portal so that their users can more easily log system issues with us and be updated on their progress, without the need for additional phone calls and emails, and it’s been decided that a convenient feature would be for it to integrate closely with our own issue tracking (which is hosted on BitBucket).

After a quick google I stumbled across a nice, albeit fairly young, API wrapper that uses RestSharp – conveniently named BitBucketSharp. It was started just a couple of months ago by Dillon Buchanan, and has made it really easy to start use the BitBucket API from within a web application.

There is another similar project, CSharp.Bitbucket, which is built upon the Spring.NET Social framework; but a quick look at the download counts on NuGet shows developers clearly favour RestSharp, with 28,230 downloads, rather than Spring.NET Social’s 529. CSharp.Bitbucket does, however, have the advantage of being hosted on NuGet, so makes it really easy to pull down and have a play with.

Since BitBucketSharp is fairly new, it currently only really has support for GET requests, so I’ve forked it and am in the process of adding support for creating/updating resources, and I’d also like to get a NuGet package up when I get the opportunity.

Check it out at bitbucket.org/j_wallwork/bitbucketsharp

<div class='box'>
    <h2 style='-webkit-user-select: none;'>Title</h2>
    <div class='boxSlide'>
        <div class='boxBg'>
            <div class='boxContent'>
                content...
            </div>
        </div>
        <div class='boxFooter'><img src='img/ws_box_footer.png'></div>
    </div>
</div>

Apologies for the above code. I realise it’s not very pleasant to look at, but unfortunately I was working on a website not too long ago which was based off of an online template, and this was the structure I’d been asked to use to add some fancy, styled boxes to a whole bunch of pages so as not to break the existing site’s css.

Now that’s quite a lot of markup for a simple section – please understand I’d be far happier using cleaner and more semantic markup, such as section, article, or even a single div if needsbe – but unfortunately that wasn’t an option. However, even if that was the code that I had to use, it sure as hell wasn’t the code I was going to write. Not over and over and over…

I first wrote a HtmlHelper extension method which would allow me to use the syntax @Html.Section(“header”, “content…”), which was definitely a marked improvement on all that code, and lead to much more readable source.

public static MvcHtmlString Section(this HtmlHelper html, string header, string content) {
   return MvcHtmlString.Create(
              "<div class='box'>" +
                  "<h2 style='-webkit-user-select: none;'>" + title + "</h2>" +
                  "<div class='boxSlide'>" +
                      "<div class='boxBg'>" +
                          "<div class='boxContent'>" +
                              content +
                          "</div>" +
                      "</div>" +
                      "<div class='boxFooter'><img src='img/ws_box_footer.png'></div>" +
                  "</div>" +
              "</div>";
}

Job done. Well, kind of. This worked well for just text-only sections. A section containing multiple paragraphs, images, or other html elements was possible, but looked a mess and was harder to read that the original syntax! It completely ruled out the option of using more complicated contents in a section – ie. if I wanted a section to contain a form. Furthermore, these boxes were nestable – so one box could potentially contain multiple. Since they took an string as content and returned an MvcHtmlString, then in addition to these problems I would have had to keep adding .ToString() to the end of each nested Html.Section().

@using (Html.BeginForm()) { ... }

How many times have we all used the above code? And what does it do? Well that’s easy, it’s an HtmlHelper for creating forms. The interesting thing with BeginForm, however, is that unlike most other HtmlHelpers, this allows for start and end tags, but with any other content nested within the curly braces (or if you prefer, open/close mustaches).

Oh what a coincidence, that sounds almost exactly like what I need to do!

So how does it work? Well fortunately, the solution is brilliantly easy. When you call BeginForm, it first outputs the opening

tag, not by returning an MvcHtmlString, but by writing directly to the view context using HtmlHelper’s htmlHelper.ViewContext.Writer.Write(string). It then returns a class implementing IDisposable (in this instance, an MvcForm), which is passed the view context when instantiated. All this class does is to write out to the view context when it is being disposed of – which as we all know is at the close of the braces.

Therefore, the code needed to allow me the freedom to write my sections as

@using (Html.Section("title here")) {
    ... any content I want! ...
}

is as follows:

public static HtmlSection Section(this HtmlHelper html, string title)
{
    html.ViewContext.Writer.Write(
              "<div class='box'>" +
                  "<h2 style='-webkit-user-select: none;'>" + title + "</h2>" +
                  "<div class='boxSlide'>" +
                      "<div class='boxBg'>" +
                          "<div class='boxContent'>"
        );
    return new HtmlSection(html.ViewContext);
}


public class HtmlSection : IDisposable
{
    private readonly ViewContext _viewContext;

    public HtmlSection(ViewContext viewContext)
    {
        _viewContext = viewContext;
    }

    public void Dispose()
    {
        _viewContext.Writer.Write(
                          "</div>" +
                      "</div>" +
                      "<div class='boxFooter'><img src='img/ws_box_footer.png'></div>" +
                  "</div>" +
              "</div>"
        );
    }
}

I use two different types of auditing in my various projects – low-level auditing, where individual property changes are recorded, and high-level, where I record actions (ie. created x, edited y). For the high-level auditing, I record each ‘action’ using a class, which inherits from a base Audit class. The basic audit class is as follows:

public class Audit : Entity {
    public virtual string CreatedBy { get; set; }
    public virtual DateTime CreatedOn { get; set; }
    public virtual Func<HtmlHelper,MvcHtmlString> Text {
        get { return html > MvcHtmlString.Empty; }
    }
}

CreatedBy and CreatedOn are both fairly obvious properties, recording when the audit was created and by whom. The Text property is used for showing the audit information in views and provides an easy and highly flexible way to display the audit text, as will be demonstrated shortly. As I said, each auditable ‘action’ is represented using a new class. These classes are subclasses of Audit, and can contain additional fields. For example, I may have an audit for the creation of a new user:

public class UserCreateAudit : Audit {
    public virtual int UserId { get;set; }
    public virtual string Username { get; set; }
    public virtual Func<HtmlHelper,MvcHtmlString> Text {
        get {
            return html => MvcHtmlString.Create(
                string.Format("Created new user {0}", Username)
            );
        }
    }
}

In order to store the audit data, we can use NHibernate’s inheritance mapping. NHibernate supports three strategies for polymorphism – table-per-hierarchy, table-per-class, and table-per-concrete-class. Table-per-class and table-per-concrete class are similar, storing the data of different subclasses in separate tables, while table-per-hierarchy stores the Audit class and any subclasses in a single table, containing all the entities in all the classes. I use table-per-hierarchy because it uses fewer joins so reading/writing is faster, although the trade off is that there will be many irrelevant columns stored for each audit type. Modern database management systems work very efficiently, minimising the effects of storing null/redundant data.

Configuring NHibernate for Polymorphism

Setting up inheritance is really easy:

public class AuditAutoMappingOverride : IAutoMappingOverride
{
    public void Override(AutoMapping mapping)
    {
        mapping.DiscriminateSubClassesOnColumn("Type");
    }
}

Ok, that’s a bit unfair. We use FluentNHibernate.AutoMapper, so if you do too then you can simply override the mapping; for everyone else, sorry but you’ll have to look it up yourself! Honestly though, it isn’t hard! I really just wanted to show you how NHibernate acheives polymorphism: it uses an extra column, in our case called ‘Type’, which acts as a discriminator. Each subclass of Audit will have it’s own discriminator, which is by default the full name of the type, and this is how NHibernate can differentiate between audit types.

Why is this good for auditing then?

There are three reasons why using polymorphism and NHibernate’s inheritance mapping strategies work well with the style of auditing that I use:

  1. Using a separate class per audit type makes it really easy to maintain the various audit types and the data relevant to each one. For instance when auditing that a user created a new task, in addition to the CreatedOn and CreatedBy fields, I want to store the task id, the type of task created, and the due date of the task; which results in a clear to read audit subclass:
    public class TaskCreateAudit : Audit {
        public virtual int TaskId { get; set; }
        public virtual TaskType TaskType { get; set; }
        public virtual DateTime DueDate { get; set; }
    }

    Also, if multiple subclasses have the same property NHibernate will aggregate them all when mapping the database table. For example if I have another audit class that has the properties DueDate and AnotherProperty, then the resulting table will have the fields Type, CreatedBy, CreatedOn, TaskId, TaskType, DueDate and AnotherProperty (note only a single instance of DueDate)

  2. The text shown when displaying the audit data is highly customizable, as a result of using separate classes for each audit type. I find using a Func<HtmlHelper,MvcHtmlString>  is a really simple, elegant, and flexible way to determine how the audit information is displayed in the view.
    public class TaskCreateAudit : Audit {
        public virtual int TaskId { get; set; }
        public virtual TaskType TaskType { get; set; }
        public virtual DateTime DueDate { get; set; }
        public override Func<HtmlHelper,MvcHtmlString> Text {
            get { return html => MvcHtmlString.Create(
                "Created new task (id #" + TaskId + ")"
            ); }
        }
    }

    Using @Model.Text.Invoke(Html) in a view will display: Created new task (id #1001). If later on I decide that actually I would rather link to the task instead, and show the due date in the audit, all I need to do is change the class definition:

    public override Func<HtmlHelper,MvcHtmlString> Text {
        get { return html => html.ActionLink(
            "Created new task due " + DueDate.ToString(),
            "Index", "Tasks", new { id = TaskId }, null
        ); }
    }

    And now the audits will display as: Created new task due 01/01/2012

  3. Finally, aside from ease of use and flexibility, auditing in this way allows for really easy querying of the data. For instance, to query all audit entries, I can use:
    _session.Query<Audit>();

    Similarly, if I want to query only a certain type of audit, I can use:

    _session.Query<TaskCreateAudit>();

In most of the projects I’ve worked on, auditing has been a fairly high level affair – typically recording user actions, such as user edited entity x or user deleted child entity y. This has been adequate for most of our systems where we do not need to be able to see exactly all the modifications made to an entity. However, for a recent system this style of auditing has been causing issues; on several occasions we’ve had requests from clients saying “a property on this entity is not as expected, but it was correct x days ago, can you see who changed it?”. And unfortunately, unless only a single user has edited the entity, we cannot see who made the change. And that really isn’t good enough…

So I took a look at NHibernate.Envers, a project which facilitates easy entity versioning with NHibernate and would therefore enable me to save an in depth history of every version of a particular entity. The docs weren’t great I found, but Giorgetti Alessandro over at PrimordialCode has a great series of posts covering virtually everything you need to know: a quick introduction, querying (part 1 and part 2) and customising the revision entity. If you’ve got some time to spare, I’d suggest you to read the introduction before continuing, as it’s a great article with some really valuable information, and clearly demonstrates the database structures generated by Envers.

 Wiring Up .Envers with Existing Auditing

After reading the PrimordialCode posts, I was able to really quickly get Envers up and running. In order to link our existing high-level action based auditing up with the detailed low-level information provided by Envers, I needed to store the revision id as part of the existing audit, which was really easy to access using IAuditReader.GetCurrentRevision(bool persist)

[HttpPost]
public ActionResult Edit(int id, UserViewModel model)
{
    var auditer = _session.Auditer();
    var user = _userBuilder.BuildEntity(model, id);
    var revisionInfo = auditer.GetCurrentRevision<REVINFO>(true);

    /* Create an instance of UserEditAudit which references the
       revision info                                             */
    _session.Save(UserEditAudit.Create(
                        LoggedInUser.UserName,
                        user, revisionInfo));

    return RedirectToAction("Overview", new { id });
}

Displaying Audit Data

Now we have the revision id for the audit, we can create a view to display the revision data. Out of the box, Envers enables you to query entities at a particular revision, and also query the revision data (to get the timestamp). Unfortunately, using the built in audit reader generates very inefficient sql with multiple and unnecessary queries.

Furthermore, I wanted to be able to show not simpy the version of the entity, but the differences between it and it’s previous version. Envers doesn’t support this behaviour out of the box, so to implement it I needed to load the two versions of the entity and diff them myself. Because of the inefficiencies of looking up entity versions using Enver’s audit reader, I wrote a generic method that could loaded the two versions of any auditable entity using a single sql query with dapper:

private IEnumerable<T> EntityAndPreviousRevision<T>(int entityId, int revisionId) where T : Entity<T>
{
    var sql = string.Format(
        @"select * from dbo.{0}_AUD where {0}Id = @entityId and REV in (
         select top 2 a.REV
         from dbo.{0}_AUD a with (nolock)
         inner join dbo.REVINFO r with (nolock) on a.REV = r.REVINFOId
         where a.{0}Id = @entityId and a.REV <= @revisionId
         order by r.RevisionDate desc )", typeof(T).Name);
    return _connectionProvider
                    .Connection
                    .Query<T>(sql, new { entityId, revisionId });
}

I then diffed the two classes by simply using reflection to iterate through each of the properties on the entity and checking for any non-matches:

public class PropertyChange
{
    public string From { get; set; }
    public string To { get; set; }

    public bool IsDifferent()
    {
        if (From == null && To == null) return false;
        if (From == null || To == null) return true;
        return !From.Equals(To);
    }
}

private static Dictionary<string,PropertyChange> DifferencesBetweenRevisions<T>(T from, T to)
{
    var properties = typeof (T).GetProperties();
    var differences = new Dictionary<string, PropertyChange>();
    foreach (var property in properties)
    {
        var pc = new PropertyChange
        {
            From = (property.GetValue(from, null) ?? String.Empty).ToString(),
            To = (property.GetValue(to, null) ?? String.Empty).ToString()
        };
        if (pc.IsDifferent()) differences.Add(property.Name, pc);
    }
    return differences;
} 

I recently needed to implement a facility on a web page to download a file that was defined by a reasonably number parameters, stored on the page as JSON. The first thing I tried was to ajax the request, but that failed as the file never downloaded – I retrieved the data, but it essentially just got ‘swallowed’.

Next I tried using JSON.stringify to populate a hidden field with the JSON value and then submit that. The action that I was posting to was defined with the parameters separately, rather than just string, and I wasn’t able to edit it: I had hoped that ASP.NET MVC 3 might have been able to decode the string but it couldn’t, so that approach failed too. (If I’d been able to amend the action then I could have obviously deserialized the string, but I wasn’t allowed [boo!]).

So in the end I wrote a quick method that would populate my form with hidden fields representing the properties of my JSON model, and that ended up working nicely. Really simple, and would obviously have to be tweaked to handle nested JSON, but it got the job done:

function downloadCurrentReport() {
    var newHiddenInputs = "", i;
    for (i = 0; i < model.length; i++) {
        var nameBase = "model[" + i + "].";
        for (var prop in model[i]) {
            newHiddenInputs +=
                    hiddenField(nameBase + prop, model[i][prop]);
        }
    }
    $("#downloadForm").html(newHiddenInputs);
    $("#downloadForm").submit();
}
    
function hiddenField(name, value) {
    return "<input type='hidden' name='" + name + "' value='" + value + "'/>";
}

Ok so I puzzled out what was wrong, and it was pretty damn annoying! It seems that the startup order was set for the web app to run before the scheduler. This meant that when the web app initialised, it looked to subscribe to the scheduler but, as the scheduler was not yet initialised, simply dropped the subscription. This meant that subsequent calls to Bus.Instance.Publish(…) did squat.

Two quick fixes:

1. Make sure the scheduler always starts up before the web app
2. Call the scheduler’s end point directly, using:

Bus.Instance
   .GetEndpoint(new Uri("msmq://localhost/scheduler_queue"))
   .Send(new UpdatedScheduledTask()
   {
       ScheduledTaskId = task.Id,
       Crontag = task.Crontag
   });

Note: probably not the best idea to hard code the uri, add it to your config instead!

So I’m currently writing a command line task scheduler so users can get various reports emailed to them at specific times. In order that the scheduler was not continuously polling the database to check for new scheduled tasks, I needed to be able to communicate from the web app to the scheduler whenever a new task was created/a task was edited, and decided to use Mass Transit for this. Mass Transit is a really easy way to set up service bus implementation, used to pass messages between parts of the system running as different processes/on different machines.

For whatever reason, I really love asynchronous stuff, it just seems so cool. So when I update a scheduled task and I see the change reflected in my command line app its pretty exciting.

When I come in to work the next day, and it doesn’t work anymore despite no code changes, it’s a real pain in the ass…

I have various forms on my application and I wanted to be able to see how much had been filled filled out, so I wrote this helper class that I could use to measure how much of the pages view model had been completed.

It uses reflection to check each of the fields in turn, simply checking whether the field is null or not. It’s a bit rough around the edges, and doesn’t really have a strategy for dealing with nullable data types (it simply ignores them) but it worked, and stopped me needing to write 10 different classes to measure 1 different view models so I was happy! It also ignores fields ending with Id, and the ignore rules can be easily extended. The other thing it does is keep track of any ignored fields, and any that it wasn’t able to count (ie. nullables). I never actually used these lists, but they come in pretty helpful during debugging I find.

public class ViewModelProgressHelper
{
    public static ViewModelProgressReport Process(TModel model)
    {
        var report = new ViewModelProgressReport();
        var fields = model.GetType().GetProperties();
        foreach (var field in fields)
        {
            if (Ignore(field.Name)) report.IgnoredFields.Add(field.Name);
            else if (field.PropertyType.IsGenericType && field.PropertyType.GetGenericTypeDefinition() == typeof(Nullable<>))
            {
                report.NumberOfFields++;
                if (field.GetValue(model, null) != null) report.FieldsCompleted++;
            }
            else if (field.PropertyType == typeof(string))
            {
                report.NumberOfFields++;
                if (!string.IsNullOrEmpty((string)field.GetValue(model, null)))
                    report.FieldsCompleted++;
            }
            else if (field.GetValue(model, null) == null)
                report.NumberOfFields++;
            else if (field.PropertyType.IsValueType)
            {
                var defaultValue = Activator.CreateInstance(field.PropertyType);
            }
            else report.UnrecognisedFields.Add(string.Format("{0} ({1})", field.Name, field.GetType().Name));
        }
        return report;
    }

    private static bool Ignore(string name)
    {
        return (name.EndsWith("Id")); // Ignore primary/foreign keys
    }
}

public class ViewModelProgressReport
{
    public ViewModelProgressReport()
    {
        IgnoredFields = new List();
        UnrecognisedFields = new List();
    }
    public int NumberOfFields { get; set; }
    public int FieldsCompleted { get; set; }
    public double Percent { get { return NumberOfFields > 0 ? ((double)FieldsCompleted / NumberOfFields) * 100 : 0; } }
    public List IgnoredFields { get; set; }
    public List UnrecognisedFields { get; set; }
}