An Introduction to Software Factories

In this article, Gunther Lenz, Microsoft ISV Architect Evangelist, introduces Software Factories from the perspective of a small to medium size software development team. He discusses the motivation for Software Factories, the basic building blocks, and how Software Factories can be realized. Furthermore, Gunther shows the potential benefits this approach has for smaller software development projects. The article is based on the experiences described in his co-authored book Practical Software Factories in .NET [2], and his work with Software Factories in industry projects.

Why Software Factories?

To answer this question we first need to understand the problems we are facing in software development projects today. The answer to these questions will help us to understand the motivation for the Software Factories paradigm. Today's text based software development can be compared to craftsmanship, rather than engineering. Therefore we can see many projects that experience massive problems keeping the schedule, staying within the budget, and delivering the promised features [3]. Even worse, many projects are never completed because of the aforementioned reasons. I believe the main reasons for this unpredictability in today’s software development projects can be traced back to:

• One-off development: Each system is virtually created from scratch. Reuse is very often ad-hoc, on very low granularity level (e.g. source code) and not in a systematic and planned way.
• Monolithic systems and increasing systems complexity: Many systems are very tightly coupled; changes in one part require changes to many other parts because of dependencies. In addition software systems become more and more complex.
• Working at low levels of abstraction: While the size of software systems and their complexity increased dramatically over the last few years, we only saw a comparatively very low increase in productivity e.g. by the move from C to C++, Java or C#.
• Process immaturity: Either a very formal approach is used, which often lacks the agility to react to changes in requirements in an efficient way, or an agile approach is used which usually does not scale very well.
• Rapidly growing demand for software systems: Over the last decade we saw a dramatic increase in demand for software. This increase in demand will continue over the next years. With the tools and techniques used today it will be very hard to meet this increasing demand.

The promise of Software Factories is to industrialize software development by exploiting economies of scope that occurs when building many similar, but yet distinct software systems.

History

The term Software Factory was first introduced in the late 1960s by R. W. Bremer of General Electric and M.D. McIlroy of AT&T in 1968. The proposed approaches differed in their goals but both can be seen as the ancestors of the Software Factories paradigm as we might know it today. The differences of the proposed approaches can roughly be summarized as: Bremer emphasized the use of standardized tools and controls whereas McIlroy focused on the systematical reuse of code when creating new software systems. The adoption of these approaches were first implemented by Hitachi, using the term ‘kojo’ (Japanese, translating to ‘factory’ or ‘works’) to label a software development facility in 1969. The reason for the adoption of Software Factories at the time was the lack of skilled labor for the software development industry in Japan.

Interestingly, while the Software Factories were based on the ideas of Bremer and Milroy, the implementations did not adhere to the exact same definition of the term Software Factory. Nevertheless, companies that implemented the Software Factories concepts experienced significant improvement in productivity, quality and process control over what they have done before.

In the 90’s it became very quiet around Software Factories until the term was reintroduced by Jack Greenfield and Keith Short with the publication of their book [1] “Software Factories, Assembling Applications with Patterns, Models, Frameworks, and Tools” in 2004. Jack and Keith describe the promise to streamline and automate software development, to become more efficient and produce higher quality software, using the Software Factories paradigm. The release of the book sparked new interest in the Software Factories paradigm. But for one or the other reason the reintroduced paradigm did not find its way into mainstream development until today. When talking to friends, colleagues, and customers I very often hear the following reasons why this might be the case:

• Software Factories are heavy weight and complex.
• Software Factories promote a Waterfall process.
• Software Factories are only for large scale development projects and not applicable to smaller product development teams, as they are often found in Independent Software Vendors (ISVs).
• Software Factories need very large scale upfront investment.
• There is no proof that Software Factories provide a ROI that justifies the investment.
• Software Factories are the same as Domain Specific Languages (DSLs).
• …

I will address these concerns throughout the discussion of this article. Let’s get started with reviewing the basic Software Factories concepts.

Building Blocks

Software Factories can be seen as a coherent methodology that is based on, and extends, 4 basic concepts:

1. Product line development: Systematic reuse is the goal using this concept in the Software Factories paradigm. What this really means is that we need to define the scope of the Software Factory that allows for systematic reuse, yet provide extension points for customizations for each product in the product line [4]. Figure 1 shows that a particular product, developed as part of the product line, contains all of the common features, some of the variable features, and some custom extensions.

   

   Figure 1: Scope, variability and extensibility of a product line

   Does this imply that Software Factories require planning the entire system upfront in a Waterfall like method you might ask? No, absolutely not. What it does mean is that usually after producing multiple similar, yet distinct products you would take a step back to think about systematic reuse. In Product Line terms this is called commonality and variability analysis. The key task is to start identifying common parts and, even more importantly, indentifying points of variability. These variability points will be filled in at the most appropriate time (e.g. design, time, implementation time, compile time, configuration time, start-up time, runtime,…) of the systems. This analysis helps us to extract functionality from the existing systems and to refactor to reusable assets that can be used as components by the product developers. This might sound like providing a framework, but in Product Line Engineering it actually means more than that. We are not just looking to reuse source code pieces, but also requirements, the architecture, configurations, scripts, test code, documentation, etc.

   In order to successfully implement such a Product Line we have to focus on the systems that we want to develop. We can only implement this systematic reuse efficiently if we define a clear and focused scope. This limited scope enables us to provide reusable components that can be efficiently be reused by the product developers. While we might start with a fairly small scope, compared to the entire software system, the scope can be extended iteratively in future releases. Certainly there is some upfront work involved, but it is not a one-shot effort at all and only an extension of the work that would need to be done even for a successful one-off development. The goal is to work in small, iterative steps, that allow the reusable assets to grow over many iterations and therefore not only limiting the upfront investment but also allowing for agile development practices. Another point I should probably mention, if you are reading some of the product line books on the market then it seems that the approach involves lots of overhead and is _very_ complex, not to say intimidating ;-). But, in reality I have seen many smaller projects (5-10 developers) using a very efficient approach to Product Line Engineering. Within these small teams a few hundred reusable function points were provided as reusable assets. Another interesting fact about those solutions was fact that the developer did not think about product line and the theory associated with it. They actually used their experience and common sense to do their development, not realizing that they were doing Product Line Engineering until we discovered the fact during our discussions.

2. Architectural Frameworks: Are not the same as development frameworks. Rather architectural frameworks provide a baseline architecture that all product line members (or in other words products of a product line) have in common. This means that we define an architecture that supports all of the products in scope of the defined product line. The architectural frameworks incorporate best practices and patterns for a particular application type, which enables more efficient development with higher quality and a uniform architecture. This might, again, sound like a one-shot upfront work to you, but in reality it is a conscious evolution and refactoring of the architecture to support all the products that are in scope through an iterative development cycle. Another important point I want to make is that we limit extensions to the product line to well defined extension points. The idea behind this is that we do not provide a generic framework that allows us to produce all possible products on this planet. Rather, we focus on a product line with limited scope in order to provide a very efficient and high quality architectural framework for the specific domain of the product line.

3. (Automated) Guidance in Context: Is based on the idea that we already implemented a few similar products before deciding to go for systematic reuse. Because we know the products and we have some examples (products or prototypes we already implemented) we can provide best practices to the developers as guidance. Actually, we can provide guidance at the time when it is needed because we extracted best practices from our initial implementations and can potentially automate the guidance that we provide. When applied in the context of a product line, guidance in context helps application developers implementing the variabilities of a particular application. The result of instructing the developer when to do what and how to do it does not only increase the quality of the product by preventing mistakes, it also leads to more efficient development and a reduced learning curve for new and/or inexperienced developers. This effect is shown in Figure 2.

   You can see that through the use of a development framework we can avoid many potential development errors through limiting the functionality provided and increase efficiency through the use of concepts on a higher abstraction level (such as WinForms rather than using the Win32 constructs). By putting guidance in context on top of a framework we can achieve an even better result because we are providing best practices and (potentially automated) guidance for the developer when needed. The result is an even narrower range for potential errors. Obviously the developer might ignore the provided guidance, but I am assuming that most developers would not do that unless they are aware of the consequences and document their deviation and the reasoning for it in the source code ;-). With the Microsoft Guidance and Automation Toolkit (GAT) it is fairly easy to provide such guidance in an iterative fashion without much investment in time and money.

   

   Figure 2: Potential implementation bugs

4. Model-driven development (MDD) and development automation: Has the goal to develop software at a much higher abstraction level using business domain concepts such as workflow, activity or message, rather than the use of only technical concepts, e.g. with using terms like class, event, library. The ultimate goal is to provide models on different abstraction levels and experts filling in the information needed at an abstraction level with concepts they are familiar with. This can be compared to building a house. Imagine models that show different aspects of the house, such as electrical, plumbing, and structural plans. In Software we could imagine the same concepts so that different stakeholders such as business analysts, project managers, infrastructure architects, solution architects, database architects, developers, and testers can use models to fill in the variability on each level. This approach allows not only for working on a higher level of abstraction, but also to verify and validate the system before implementation. Code generators fill the gap between the high-level modeling language and "low-level" code. These specialized modeling languages are also called Domain-Specific Languages (DSL). Software development not only becomes more efficient but it also enables stakeholders without programming experience to participate and contribute to the development cycle. The rule of thumb for DSL efficiency is: “The more specialized (smaller) the domain for a DSL, the more efficient the DSL”. Compare this to the WinForms DSL in Visual Studio. The DSL is restricted to the domain of being GUIs, but it is very efficient for that domain compared to let’s say .NET. Figure 3 shows the concept of multiple, interrelated, DSLs enabling working on different abstraction levels.

   

   Figure 3: Interrelated DSLs on multiple abstraction levels

   When talking about DSLs I very often hear the objection that this is not new, and that case tools promised that same in the mid 80’s but the tools failed miserably delivering the promised value. IMHO there are many reasons why case tools did not live up to their promise and I don’t want to reiterate over all of them since this would fill another 10 pages easily. Nevertheless, I want to highlight some of the main reasons why I think case tools did not succeed. First, case tools were working on a very high abstraction level (like the Requirements DSL in Figure 3). Second, case tools did not restrict the domain but rather tried to be able to model all the systems in the world (this is way in contrast to the mentioned thumb for DSLs). Third, the tools only used one model on a very high abstraction level to fill in most of systems variabilities. Therefore, the code generators had to make many assumptions about the system and this lead in many cases to large amounts, inefficient and bloated source code generation. Another comment I hear very often when talking about DSL’s is that it takes too much time and effort to develop a DSL. Well, with the availability of modern modeling tools such as the Visual Studio DSL Toolkit, Eclipse, and MetaEdit it is much easier and faster to define and implement custom DSLs that fit the needs of the targeted domain and stakeholders than it used to be. I recommend building a small prototype to get a feel for the tools and possibilities, before jumping to conclusions ;-). Certainly, the development of efficient DSLs require very good domain knowledge and some practice. Actually you can see some new graphical DSL’s in Visual Studio such as the Distributed Systems and Class Designers.

   Combining the Pieces

   Software Factories combine the four building blocks, extend, and interweave them (as shown in Figure 4) to build a coherent software development paradigm.

   

   Figure 4: Software Factories building blocks

   For example an architectural framework can provide extensibility points required for product line development. Another example is developer guidance provided to the developers at implementation time within the IDE at the time and place when it is needed. The guidance could suggest what to do next e.g. to create a data access layer, and provide information on how to do it, or even better, provide automation that generates source code by asking the developer to provide input to fill in variability points, such as database information for example.

   Software Factory Definition

   In the remaining part of this article I discuss the pieces that make up a software factory, and how we define it in our book. The key deliverables for a Software Factory are the Software Factory Schema and the Software Factory Template as shown in Figure 5. You can see the two roles of software architect and application developer. The Software Architects are usually responsible defining and implementing the two software factories deliverables, namely the Software Factory Schema and the Software Factory Template.

   The Software Factory Schema is a description of how to implement the products that can be produced with this Software Factory. The Software Architects define and implement a collection of core assets that make up the common and variable features of a product line and provide extension points to add additional functionality. The Software Factory Schema describes the assets, artifacts, and activities necessary to produce the applications in scope of the Factory. You can compare this to a recipe for baking a cake. The Software Factory Template is the implementation of the Software Factory Schema usually packaged in one or multiple installation packages. The Software Factory Template then is installed on the application developer’s computer. Therefore, the Software Factory becomes a part of the application developer’s development environment and enables him to develop instances of the software product line more effectively and predictably. Typical examples for the content of the Software Factory Template are: Visual Studio templates, providing starting points for developing projects, reusable component blocks to build the solution, guidance on how to build solution and which activities to perform when, DSLs, requirements, architecture frameworks, application blocks, other 3rd party components, and documents that application developer will use when implementing an instance of a product line are typical examples of the Software Factory Template content . You can compare the Software Factory Template to the ingredients you need to bake the cake described in the recipe.

   

   Figure 5:  Software Factory deliverables

   While in larger organizations, the roles of architect and developer are usually filled by different people, in smaller companies I often see both roles filled by the same persons wearing multiple hats.

   While there are certainly a lot more details that I could describe, these are the basic concepts necessary to understand the concepts that make Software Factories work as well as how to implement them. In my conversation with different people it turned out that some smaller ISVs actually implemented Software Factory like solutions. This should not be too surprising since Software Factories builds on concepts that are proven as well as on best practices collected throughout the industry. Once more I would like to emphasize, that developing a Software Factory, as any other software development, should be done in an iterative way and the Software Factory Schema and Software Factory Template will be evolved over many development iterations.

   Quantitative Benefits

   Figure 6 shows the quantitative ROI for a project we implemented using the Software Factories paradigm. In the given scenario I implemented one DSL, and a few guidance packages for the customer (using the Microsoft DSL Toolkit and the Microsoft Guidance and Automation Toolkit (GAT)). You can see that there is some initial investment associated with the implementation of the Software Factory. Nevertheless, after producing ~8 products using the factory we reach the point of break even. In the given scenario the plan was to produce at least 20 products with the factory which cuts the cost, compared to one off development, almost in half! Again, the initial investment for the software factory was substantial on this case, compared to one-off implementations, but the result is a much lower marginal cost for each additional product.

   

   Figure 6:  ROI of using the Software Factories paradigm

   We did not consider non measurable positive effects such as improved product quality and maintenance in this calculation. If you want more information on this scenario then check out the msdn article at http://msdn.microsoft.com/en-us/library/cc496679.aspx we wrote.

   Liabilities

   Of course Software Factories do not come without any downsides and responsibilities. The most common are the following related liabilities:
   • Development of a Software Factory takes resources and time. The cost of developing a complex factory can be significant. In most cases the factory can be evolved over multiple iterations to keep the initial investment lower (and to enable harvesting existing products for common assets). In order to justify this investment it will have to amortize itself over a number of instances of the product built using the Software Factory.
   • Developing a Software Factory requires strong domain expertise, which is usually based on experience with building software systems in the same business domain. A factory is a generalization of the assets and the best practices harvested from previously built applications. It takes the foresight and the discipline to the explicit objective of a project to create assets and guidance for an emerging Software Factory.
   • The factory scope needs to match the requirements of all the applications that are planned to be built. This liability has two distinct sides. On the consumer side a misapplied factory can bring more problems than benefits. More variability allows better matching of requirements, which on the other hand increases the initial investment.

   Before investing into a Software Factory, the pros and cons have to be carefully evaluated. There are a number of factors that are clearly in favor of building a factory: A narrow business domain with known variabilities, expertise in building applications in a particular domain, or existing application allow for factoring out architectural assets and best practices, just to name a few.

   Conclusion

   In this article I introduced the motivation for the Software Factories paradigm. I then showed the building blocks that build the foundation for Software Factories and how these concepts are being interwoven and extended to form a coherent software development paradigm. After that I showed the impressive quantitative results of a Software Factories project I implemented for a customer and closed with some words of caution when planning on using the paradigm. Interestingly the evaluated project was rather small, and the team working on the Software Factory consisted of 1 to 4 persons at a time. The results show that even, or better especially, with minimal investment we achieved a very impressive result. The overall ROI was ~100% if calculated over the planned 20 products to be implemented with the factory. Obviously Software Factories are not a silver bullet, and they do not fit all projects and or situations. Nevertheless, I believe that Software Factories have tremendous potential and can provide a viable option for smaller teams if carefully planned and implemented.

   About the Gunther Lenz

   Gunther recently joined Microsoft in the role of ISV Architect Evangelist. His goal is to enable ISVs to efficiently architect, implement and deploy software solutions by fully, and efficiently, leveraging the potential of the Microsoft Development Platform. Before joining Microsoft, Gunther worked as a senior consultant on software architecture and software factories at Siemens Corporate Research in Princeton, NJ. With more than 10 years of experience in software architecture, design, implementation, and deployment of complex enterprise software solutions, Gunther has in-depth experience in the entire Software Development Lifecycle. As a result of his work, he published two books (".NET -- A Complete Development Cycle" and "Practical Software Factories in .NET") as well as numerous articles in magazines. Gunther is also an invited speaker at international conferences and received the Microsoft Most Valuable Professional (MVP) - Solution Architect award in 2005/2006/2007. Visit Gunther's blog at http://blogs.msdn.com/glenz and http://blogs.msdn.com/usisvde/.

   
   

   References

   [1] Jack Greenfield, Keith Short, Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools (Wiley, 2004, ISBN 0471202843)

   [2] Gunther Lenz, Christoph Wienands: Practical Software Factories in .NET (APress, June 2006, ISBN 159059665X)

   [3] Paul Clements, Linda Northrop, Software Product Lines – Practices and Patterns (Boston, MA: Addison Wesley, 2002, ISBN 0201703327)

   [4] IEEE-SA Standards Board, IEEE Recommended Practice for Architectural Description of Software-Intensive Systems (IEEE 1471-2000, ISBN 0-7381-2519-9)

   [5] Gunther Lenz, Christoph Wienands: Making the Business Case for Software Factories, http://msdn.microsoft.com/en-us/library/cc496679.aspx