Map-Reduce, Python and Named Tuples

A year and change back, I wrote this on "Exploratory Programming".It turns out that it was a mistake. While the subclass-expansion technique is a cool way to bang out a program incrementally, in the long run, the subclassing is ill-advised.

The more I look at Python generator functions and the idea of using Map-Reduce, the more I realize that Visitors and Subclass Extension are not the best design pattern. The same kind of exploration can be done with map-reduce techniques, and the resulting application is slightly simpler.

Design Coupling

The problem with design-by-subclass is that the map and reduce operations are often defined relatively informally. After all, they're just method invocations within the same class. You can, unthinkingly, create obscure dependencies and assumptions. This can make the Visitor rather complex or make subclass features hard to refactor into the Visitor.

As a concrete example, we have an application that processes directories of files and file archives (ZIP and TAR) of workbooks with multiple sheets. All of this nesting merely wraps a source of spreadsheet rows. This should be a collection of simple nested map operations to transform directories, files, archives of files, etc., into a spreadsheet row source.

Part way through our class hierarchy, however, a subclass introduced a stateful object that other method functions could use. However, when we tried to refactor things into a simple Visitor to visit all of the workbooks (ignoring all the directory, archive and file structure), we worked around the hidden stateful object without realizing it.

Named Tuples and Immutability

A much cleaner solution is to make use of Python's namedtuple constructor and write generator functions which map one kind of namedtuple to another kind of namedtuple. This has the advantage that -- unless you've done something really bad -- you should be able to pickle and unpickle each tuple, easily seeding a multi-processing pipeline. Maximal concurrency, minimal work.

Each stage in a map-reduce pipeline can have the following form.

SomeResult = namedtuple('SomeResult',['a','b','c'])
for x in someSource():
  assert isinstance( x, SomeSource )
  yield SomeResult( <i>some transformation of x</i> )