Complexity is the Problem

Who really believes that complexity is not a problem? The complexity of modern software is a problem. Some say that it might be THE problem.

Let's say that you are building an Internet facing web application using a Java-based technology stack. (Pick one.) Building a modern application with a Web 2.0 front-end and web services back-end, add connection to a database and object store. Add security and testing and source control management, poly-oriented methods of development. Who thinks that this is easy?

And then, we have the latest recommendation for how we ought to think about the problem. We have been hardware-oriented, operating system-oriented, platform independent-oriented, language-oriented, procedure-oriented, dynamic-link-oriented, process-oriented, management-oriented, user-oriented, data-oriented, information-oriented, test-oriented, function-oriented, version-oriented and now module-oriented. In general, we keep on trying to figure out how to deal with overwhelming complexity.

Think of the short history of the software industry. The individual, teams, companies and the whole industry looks for ways to reduce the complexity of some problem. Do you remember a time when we were unable to run the same program on two different versions of the same operating system? Backward compatibility became an important aspect of the operating system itself. Younger ones might take it for granted. Or, do you remember when we were unable to compile the same program on two different operating systems? Platform-independent source code became a tradition for compiling a program. Younger ones might think that the solution is obvious. Do you remember when two computers could only communicate with a point-to-point connection? Packet-switched networking became an important aspect of simplification. Younger ones may not be able to imagine a world without networking. Do you remember when binary-identical programs were unable to run on two different operating systems? An abstract machine, such as i5/OS and Java, became an important aspect of simplification. Remember when a database library was embedded within a program? Remember when a browser could only display static pages?

Our goal is to reduce complexity to a level we can manage. Complexity must be managed. Managing it is an aspect of all construction. Techology must be applied on a case by case basis. Adding a technology when it is not required increases complexity. We might say that every development effort is an effort to simplify something, just as every tool is an effort to simplify something. But simplifying one thing likely makes other things more complex.

Complexity is a problem that cannot be "solved" in the same way that a puzzle can be solved. Over the years, software in general has become increasingly complex. When software becomes so complex that we can no longer manage, we look for ways to reduce its complexity back to something managable. Therefore, complexity is never going to go away. It will always be THE problem.

OSGi is "the dynamic module system for Java." It tries to solve one basic, recurring and complex problem for a Java-based application: the java.class.path property (set by the classpath and cp parameters). The problem is basic because the standard java.class.path mechanism does not scale. It is not possible to put thousands of jars on the java.class.path in a platform-independent way because there are hard limits to the length of a command line. (I work with an application that has more than 4,000 jars and an operating system that has a hard limit of 4,000 characters on the command line. It just doesn't fit.)

The problem is recurring because every developer must eventually deal with multiple versions of an application and its dependencies. The standard java.class.path mechanism does not account for different versions of a class, package, resource or framework within one instance of a virtual machine. When java.class.path does not work, where do we go from here?

The problem is complex because some well-known Java mechanisms do not even work with multple class loaders. In other words, a method that works when running with java.class.path may not work when loaded by a custom class loader. A developer must learn to stop using mechanisms that do not work in the new environment.

Going from monolithic (non-modular) to modular is inherently complex. When you have decided to go modular, the question becomes: Are you going to adopt an existing module system, such as OSGi, or build your own? A module system with less complexity than OSGi is possible. I myself have built not one but three. But OSGi offers shared knowledge with an industry standard, multiple implementations of the framework, off-the-shelf collection of pre-compiled modules. Besides, smart people are working to make OSGi less complex.