Pipes and Filters Pattern in .NET

A pipeline in software context is a very well-known architectural style in which a process consists of a series of steps to be followed  in order to proceed the data, and the output of one step is the input of another step. This is also called the Pipes and Filters design pattern. The naming comes from the physical pipeline as this architectural style is very similar to a pipeline in which a stream of data comes in and leaves after being processed. The original idea of pipeline in software is implemented in Unix.

This pattern is used in many places. Compiler pipeline, ASP.NET HTTP Pipeline, and workflows are three of many examples that I can mention.

The pipes and filters style is implemented in various platforms with different techniques and technologies. Recently I was in a situation to implement this pattern and did some research to find more about the possible options to implement this pattern in the .NET Framework.

Doing my research, I found many approaches introduced by community members but the most mature technique is the one that Oren Eini has described in his blog post using Generics. There is also an interesting technique described by Jeremy Likness using the yield keyword in C#.

In this post I’m going to apply Oren’s approach and expand it to write a simple implementation of the classic KWIC example in Software Engineering. I liked Oren’s code because as he said, it’s comparatively simpler than other solutions introduced for this problem in the .NET Framework.

An Overview of KWIC

KWIC stands for Key Word in Context and is a classic problem in Software Engineering papers in which you try to create an index of words by sorting and aligning each word in a piece of text. David Parnas has a famous paper on modularity that uses KWIC as an example.

There are some basic and advanced implementations of KWIC in different platforms but the main steps are:

• Reading the input
• Shifting the words in each line to get a new permutation
• Sorting the results
• Writing the output

Interestingly, in this case the output of each step is the input of the next step which makes this a good candidate for the Pipes and Filters pattern.

Implement the Pipes and Filters Pattern with Generics

Oren’s technique for implementing the Pipes and Filters in the .NET Framework is based on a Generic interface and a Generic class. The Generic interface simulates the filter and the Generic class simulates the pipeline.

The IOperation interface has a single method called Execute that is the implementation of the filter logic. Each filter should implement this interface.

using System.Collections.Generic;namespace KwicPipesFilters{    public interface IOperation<T>    {        IEnumerable<T> Execute(IEnumerable<T> input);    }}

The use of a generic IEnumerable is a good choice because it leaves a lot of space for the developers to plug in any type that they want and use various types for their filters.

The Pipeline class has an Execute and a Register method. Using the Register method, you add different filters to the pipeline and using the Execute method, you start processing the item in all the registered filters.

using System.Collections.Generic;namespace KwicPipesFilters{    public class Pipeline<T>    {        private readonly List<IOperation<T>> operations = new List<IOperation<T>>();        public Pipeline<T> Register(IOperation<T> operation)        {            operations.Add(operation);            return this;        }        public void Execute()        {            IEnumerable<T> current = new List<T>();            foreach (IOperation<T> operation in operations)            {                current = operation.Execute(current);            }            IEnumerator<T> enumerator = current.GetEnumerator();            while (enumerator.MoveNext());        }    }}

The implementation of the Pipeline class is straightforward: it keeps a list of filters and provides a Register function that lets you add new filters to your pipeline, and then use the Execute method to execute all the filters in the list to process an input.

Reader

The Reader filter reads the input text from a file and returns an IEnumerable list of lines. Of course, for the first filter in the pipe we don’t care about the input as the input is read inside the filter itself.

using System;using System.Collections.Generic;using System.IO;namespace KwicPipesFilters{    public class Reader : IOperation<string>      {        public IEnumerable<string> Execute(IEnumerable<string> input)        {            Console.Title = "Pipes and Filters Pattern in .NET";            Console.WriteLine("Enter the path of the file:");            return File.ReadLines(Console.ReadLine());        }    }}

Shifter

The Shifter filter is where the main logic of the KWIC application is implemented. It shifts the words in each line to find all the possible permutations suitable for the index.

using System.Collections.Generic;namespace KwicPipesFilters{    public class Shifter : IOperation<string>      {        public IEnumerable<string> Execute(IEnumerable<string> input)        {            List<string> shifts = new List<string>();            foreach (string line in input)            {                string[] words = line.Split(new char[] { ' ' });                for (int i = 0; i <= words.Length - 1; i++)                {                    shifts.Add(string.Join(" ", words));                    string firstWord = words[0];                    for (int j = 1; j <= words.Length - 1; j++)                    {                        words.SetValue(words[j], j - 1);                    }                    words.SetValue(firstWord, words.Length - 1);                }            }            return shifts;        }    }}

Here we have a basic implementation of the Shifter filter where we split the line into separate words based on the space between them, then shift all the words to find various permutations.

Sorter

Before returning the final results in the Writer filter, we need to sort the index alphabetically. This is done in the Sorter filter.

using System.Collections.Generic;using System.Linq;namespace KwicPipesFilters{    public class Sorter : IOperation<string>    {        public IEnumerable<string> Execute(IEnumerable<string> input)        {            LineComparer lineComparer = new LineComparer();            input.ToList<string>().Sort(lineComparer);            return input;        }    }}

Here I used a LineComparer class to implement the ICcomparer interface for the string type.

using System.Collections.Generic;namespace KwicPipesFilters{    public class LineComparer : IComparer<string>    {        public int Compare(string x, string y)        {            return string.Compare(x, y);        }    }}

Writer

Obviously, the last filter should write the index to the output for the user and that’s the purpose of the Writer filter.

using System;using System.Collections.Generic;namespace KwicPipesFilters{    public class Writer : IOperation<string>    {        public IEnumerable<string> Execute(IEnumerable<string> input)        {            foreach (string line in input)            {                Console.WriteLine();                Console.WriteLine(line);            }            Console.ReadLine();            yield break;        }    }}

As you see, this filter uses a yield break to avoid returning any result.

Pipeline

Having all the filter implemented, I also need to implement the pipeline itself in order to register the filters and make the whole thing work. I do this in my KwicPipeline class with a simple code that it has.

namespace KwicPipesFilters{    public class KwicPipeline : Pipeline<string>    {        public KwicPipeline()        {            Register(new Reader());            Register(new Shifter());            Register(new Sorter());            Register(new Writer());        }    }}

I inherit from the Pipeline class and register my filters in the public constructor.

Putting It Together

There is only one step remained and that is putting all these things together to start the pipeline. All I need to do is to create an instance of the KwicPipeline class, call its Execute method, and leave the rest to my pipes and filters.

namespace KwicPipesFilters{    class Program    {        static void Main(string[] args)        {            KwicPipeline pipeline = new KwicPipeline();            pipeline.Execute();        }    }}

Conclusion

In this post I implemented the Pipes and Filters pattern in the .NET Framework using a simple and generalized technique that relies on Generics to implement the KWIC application. In my opinion this is one of the best ways to implement this pattern in the .NET Framework. I have uploaded the sample source code package here. Note that the solution is created using Visual Studio 2010 RC1.

There are other techniques to implement this pattern in .NET and one specific technique that I have in mind is using the Windows Workflow Foundation. I may work more on this idea and write about it later.