Designing a system with DSLs

This article is taken from Building Domain Specific Languages in Boo from Manning Publications. It introduces key DSL concepts and shows how they work with an in-depth message-routing example. For the table of contents, the Author Forum, and other resources, go to http://manning.com/rahien/.

It is being reproduced here by permission from Manning Publications. Manning early access books and ebooks are sold exclusively through Manning. Visit the book's page for more information.

MEAP Began: February 2008
Softbound print: November 2009 (est.) | 400 pages
ISBN: 1933988606

 

Use code "dzone30" to get 30% off any version of this book.

Business DSL is almost always about rules and the actions to be taken when those rules are met.

This sounds like a very narrow space, doesn't it? But let me state in another way, the place of a business DSL is to define policy, while the application code defined the actual operations. A simple example will be defining the rules for order processing. Those, in turn, will affect the following domain objects:

• Discounts
• Payment plans
• Shipping options
• Authorization Rules

The application code then takes this and acts upon it.

Policy is usually the place where we make most changes, while the operations of the system are mostly fixed. We are also not limited to a single DSL per application, in fact, we will probably have several, both technical and business focused DSLs. Each of those will

handle a specific set of scenarios (processing orders, authorizing payments, suggesting new products, etc).
What about building the entire system as a set of DSLs?

That may be a very interesting approach to the task. In this scenario, we inverse the usual application code to DSL metrics, and decide that the application code that we would like to build would be about mostly infrastructure concerns and the requirements of the DSL. I would typically like to use this type of approach in backend processing systems. Doing UI on top of a DSL is certainly possible, but at this point, I think that we will hit the point of diminishing returns. Good UI are complex. A complex DSL is a programming language, and at the point, you would probably want to use a programming language to work with rather than a DSL.

There is an exception to that, assuming that your application code is in Boo. Then you are working with a programming language, and then you can build a technical DSL that will work in concrete with the actual frameworks that you are using. Rails is a good example of this approach.

Assuming that you write your application code in a language that is not suited for DSL, however, you would probably want to have a more strict separation of the two approaches.

My own feelings on the subject state that the best approach for that would be to use a DSL to define policy and the application code to define the framework and the operations that can be executed by the system.

Building such a system turns out to be almost trivial, because all you need to do is apply the operations (usually fairly well understood) and then you can play around with the policy at will. If you have done your job well, you'll likely have the ability to sit down with the customer and define the policy and have them review it at the same time. Here we will use a simple example: a DSL for routing and translating messages.

Designing the message routing DSL

Consider the following situation, in a backend processing system; we have decided to use messages as the primary means of communication. Several external systems will communicate with the order processing system. Among them is a web application, business partners, administration tools and the company’s warehouse system.

All of those are built by different teams, according to different schedules, priorities and constraints. Our system is the big black hole in the middle, around which all the others are orbiting.

As such, it will not surprise you in the least to learn that we have chosen messaging as the means to communicate with the outside world. This approach has many benefits, versioning and scalability not the least of them, all of which have been exhaustively outlined elsewhere. I’ll just say that this was the chosen approach, and let it end there.

That brings us the problem of how to take an incoming message and dispatch it to the correct handler in the application. Message translation and routing is a very simple domain, but it usually looks fairly nasty in code.

This is especially true if we want to take versioning into consideration, or if you want to deal with heterogeneous environments.

Let us start from the simplest alternative, we have an end point that can accept JSON formatted messages, and process them. We will take a peek at the big picture first, in figure 1.

Figure 1 – Routing messages using DSL

We start from some external application that sends a JSON message to a given end point. This endpoint will take the JSON string and translate it to a JSON Object, then pass it to the routing module.

The routing module will use a set of DSL scripts in order to decide how to route a specific message to the business logic. The business logic will perform its job, and can return a reply, which will be sent to the client.
So far, it is fairly typical messaging scenario. We only need to add asynchronous messages and we can call ourselves enterprise developers.

Now, imagine that you are in a movie, and we slowly zoom on the routing / DSL part of figure 1. As we zoom down, we can see more and more details...

Well, you could, if we had any. I guess we really need to build something here. How are we going to start?

By specifying that the responsibilities of the routing modules are:

• Accepting messages in a variety of formats (XML, JSON, CLR Objects, etc)
• Translating a message from external representation to internal representation
• Dispatching an internal message to the appropriate handler

We know what we need to build; we are left with deciding on the syntax. This DSL is meant for technical people, most probably the developers on the project. The main reason that we want to use a DSL here is to keep it flexible and easy to add new messages and transformations.

This, in turn, means that we can use a technical DSL here. It will probably have the following structure:

• Can this script handle this message?
• Transform the message to the internal message representation
• Decide where to dispatch it.

Implementing the message routing DSL

With that in mind, we can start writing a draft of the syntax, which appears on listing 1.

Listing 1 – Initial draft of the Routing DSL

# decide if this script can handle this message
return unless msg.type == “NewOrder” and msg.version == “1.0”

# decide which handle is going to handle it
HandleWith NewOrderHandler:
        # define a new list
        lines = []
        # add order lines to the list
        for line in msg.order_lines:
                lines.Add( OrderLine( line.product, line.qty ) )
        # create internal message representation
        return NewOrderMessage(
                msg.customer_id,
                msg.type,
                lines.ToArray(OrderLine) )

This looks like a good enough place to start from, right? It is straightforward to read and to write, and it satisfy all the requirements that we had. It is a highly technical DSL, but that is fine, because we are aiming for technical people.

You may note that it is very easy to create techincal DSL, because you don’t have to provide a lot of abstraction over what the language offer. You mostly need to provide a good API and good semantics.

Let us get to the implementation, shall we?

We will start with the routing part. how do we get the messages in the first place? We need some way to handle several types of messages types, and do it without place undue burden on the developers. After all, we intend to have a lot of messages, and having to deal with writing adapters or translators for them is exactly why we went with the DSL route.

However, we also want to keep our DSL implementation as simple as possible. If I need to do things like xmlDocument.SelectNodes(“/xpath/query”) in the DSL on a routine basis, I probably have an abstraction issue somewhere. Let us take a look at figure 2, which shows how we can solve this issue:

Figure 2 – The Router class

As you can see, we have a single method here, Route()(1) that accept an IQuackFu. It allows us to handle unknown method calls at runtime, in a smart fashion. We used it to build the XMLObject before, and in this case, we separate the implementation of the message from its structure.

This means that I don’t care if the message is XML, JSON or a plain CLR object. I can treat as if it was standard object, and let the IQuackFu implementation to deal with the implementation details.

This keeps maximum flexibility with a minimum of fuss all over the place.

Once that is solved, we get down to the actual building of the DSL. We are going to use Rhino DSL here, to take care of all the heavy lifting of the DSL building.

We will start with the typical first step, defining the implicit base class that will be the basis of our DSL.

Implicit Base Class

The implicit base class is one approach to building a DSL. It is composed of a base class that we define in our application code, and a compiler step in Boo that will turn the DSL script into a class that is derived from the defined base class. Hence, the base class moniker.

The implicit part of the name comes from the fact that in the DSL script itself, there is no reference to that class, it is implicit.

There are three major advantages to this approach. The first is that we can now refer to DSL script instances using the base class, by utilizing standard OOP principals. The second is that the base class can expose method and properties that are useful in the DSL. This means that the base class itself composes part of the language that we are creating.

The last advantage for using implicit base classes is somewhat strange. If the class is implicit, we can replace it. This is extremely helpful when we want to test a DSL or version it.

The idea of Implicit Base Class allows us to define the language keywords and constructs very easily.

Listing 2 has the entire code of the base class.

Listing 2 – The base class for the Routing DSL

/// <summary>
/// This delegate is used by the DSL to return the
/// internal representation of the message
/// </summary>
public delegate object MessageTransformer();

public abstract class RoutingBase
{
    protected IQuackFu msg;
    public string Result;

    public void Initialize(IQuackFu message)
    {
        msg = message;
    }

    /// <summary>
    /// Routes the current message. This method is overriden by the
    /// DSL. This is also where the logic of the DSL executes.
    /// </summary>
    public abstract void Route();

    public void HandleWith(Type handlerType, MessageTransformer transformer)
    {
        IMessageHandler handler = (IMessageHandler) Activator.CreateInstance(handlerType);
        Result = handler.Handle(transformer());
    }
}

So, how does it work? The DSL script goes into the Route() method, and the msg field contains the current message. When we execute the Route() method, the DSL code is executed. The second line on listing 1 checks to see if the message is a match, and if it isn’t, is simply return without performing any action.

Then we have the HandleWith NewOrderHandler and the code beneath that. Here we are using Boo’s ability to infer things for us. In this case, we pass the type name as the first parameter, and Boo will turn that into a typeof(NewOrderHandler) for us. The code underneath the HandleWith line is using implicit blocks to pass the delegate that will transform the message to its internal representation.

We now need to have a way to compile this DSL. We do it using a DSL Engine(2), listing 3 shows how this is done:

Listing 3 – The Routing DSL Engine

public class RoutingDslEngine : DslEngine
{
    protected override void CustomizeCompiler(
            BooCompiler compiler,
            CompilerPipeline pipeline,
            string[] urls)
    {
        // The compiler should allow late bound semantics
        compiler.Parameters.Ducky = true;
        pipeline.Insert(1,
            new ImplicitBaseClassCompilerStep(
                // the base type
                typeof (RoutingBase),
                // the method to override
                "Route",
                // import the following namespaces
                "Chapter4.MessageRouting.Handlers",
                "Chapter4.MessageRouting.Messages"));
    }
}

And that is it; we have our DSL ready to roll, almost.

Oh, we need to hook it up to the Router class, right? We will first look at the code in Listing 4 and then discuss it.

Listing 4 – The Router class handles message dispatch for the application

public static class Router
{
    private static readonly DslFactory dslFactory;

    static Router()
    {
        dslFactory = new DslFactory();
        dslFactory.Register<RoutingBase>( new RoutingDslEngine());
    }

    public static string Route(IQuackFu msg)
    {
        StringBuilder messages = new StringBuilder();
        RoutingBase[] routings = dslFactory.CreateAll<RoutingBase>(Settings.Default.RoutingScriptsDirectory);

        foreach (RoutingBase routing in routings)
        {
            routing.Initialize(msg);
            routing.Route();

            if (routing.Result != null)
                messages.AppendLine(routing.Result);
        }

        if(messages.Length==0)
        {
            return "nothing can handle this message";
        }

        return messages.ToString();
    }
}

We have quite a few concepts to discuss here. In the constructor, we are creating a new DSL Factory and registering our Routing DSL Engine, but the real thing happens in the Route(msg) methods.

We ask the DSL factory to give us all the DSL instances in a specific folder. This is a nice way of handling a set of scripts (note that it tends to break down when you have more than a few dozen scripts; at that point, you want better management of them.

We get back an array of RoutingBase instances, which we just iterate over and run. This lets all the scripts to handle a shot at handling this message.

The last piece we are missing is the JSON Endpoint and the JsonMessageAdapter. We will start from the end point, since this is simple ASP.Net stuff. We create an Http Handler class to accept the messages, and then send them to be routed. Listing 5 shows how it is done:

Listing 5 – The JSON End point

public void ProcessRequest(HttpContext context)
{
    //verify that we only allow POST http calls
    if (context.Request.RequestType != "POST")
    {
        context.Response.StatusCode = 400;
        context.Response.Write("You can only access this URL using POST");
        return;
    }

    // translate from the post body to a json object
    byte[] bytes = context.Request.BinaryRead(context.Request.TotalBytes);
    string json = Encoding.UTF8.GetString(bytes);
    JsonSerializer jsonSerializer = new JsonSerializer();
    JsonReader reader = new JsonReader(new StringReader(json));
    JavaScriptObject javaScriptObject = (JavaScriptObject)jsonSerializer.Deserialize(reader);

    // send the json object to be routed
    string returnMessage = Router.Route(new JsonMessageAdapter(javaScriptObject));
    context.Response.Write(returnMessage);
}

The code deals mostly with unpacking the data from the request, de-serializing the string into an object. The really important thing happens on the line before last. We call Router.Route() and we pass a JsonMessageAdapter.

This class is responsible for translating the JavaScriptObject into an IQuackFu that we expect in the Routing DSL.

The code for JsonMessageAdapter is in listing 6:

Listing 6 – The JsonMessage Adapter implementation

public class JsonMessageAdapter : IQuackFu
{
    private readonly JavaScriptObject js;

    public JsonMessageAdapter(JavaScriptObject js)
    {
        this.js = js;
    }

    public object QuackGet(string name, object[] parameters)
    {
        object value = js[name];
        JavaScriptArray array = value as JavaScriptArray;

        if(array!=null)
        {
            return array.ConvertAll<JsonMessageAdapter>( delegate(object obj)
                {
                    return new JsonMessageAdapter( (JavaScriptObject) obj);
                });
        }

        return value;
    }
}

We are only showing the QuackGet() methods here, QuackSet() and QuackInvoke() are not implemented.

About the only interesting thing here is how we deal with arrays, since we need to convert them, as well, to JsonMessageAdapter arrays.

And… that is it all, folks. Honest. We need around 200 lines of code to build this. And it takes about an hour or so.

Go back to listing 1 and look at the DSL that we wrote. We can now use it to process JSON messages like the one on listing 7.

Listing 7 – A JSON Message that can be handled by our DSL

{
    type: "NewOrder", version: “1.0”,
    customer_id: 15,
    order_type: "fast",
    order_lines:
        [
            { product: 3, qty: 5 },
            { product: 8, qty: 6 },
            { product: 2, qty: 3 },
        ]
}

Extending this infrastructure to deal with XML objects, for example, is simply a matter of creating an XmlMessageAdapter and adding a new end point that can accept it.

We had, in the space of a few pages, created a DSL, implemented the structure around it and are now ready to put it to some good use. It wasn’t very complex, and we didn’t even have to use any of the advanced options that are at our disposal.

The reason it was so simple is mostly that we can get away with having a very technical language here, so we just utilize the built-in syntactic sugar that we have in Boo to get a nice DSL, but not much more. Nearly everything we did here was infrastructure code and dealing with running the DSL.

(1) One thing to note here, we are using a string as the return type. In real world scenarios, we would probably want to return a message as well.

(2) DSL Engine – part of the Rhino DSL set of tools. A DSL Engine contains the configuration required to change Boo into our DSL.