Macros, Preprocessors and DSL Development

Along with the recent trend of DSLs becoming more and more popular, we are also seeing a growing trend of programming languages adding preprocessing and macro based features as part of their machinery. Is this a mere coincidence or we are becoming more aware towards Guy Steele's words of wisdom that "a main goal in designing a language should be to plan for growth".

Compile time meta-programming has long been dominated between the 2 extremes of C pre-processors and Lisp macros. In the context of DSL implementation, I have been doing some reading on syntax extension features and meta-programming in various languages. Even I came across this thread in the core-ruby discussion group, where people have been talking about implementing Converge style macros in Ruby. Lisp and Dylan implement macros mainly on top of a language that's syntactically minimal. But nowadays, we are looking at syntax rich languages like Template Haskell and MetaOCaml that implement macros as part of the language.

Converge is, of course a very interesting experiment, where Tratt has implemented Template Haskell like macro capabilities on top of a Python like dynamically typed language. Converge macros are different from Lisp, in the sense that unlike Lisp, they implement macro calls as a special syntax, while macro definitions are regular functions. When the compiler encounters the special syntax in a macro call, it does relevant processing for the quasi-quotations and splice annotations and builds up the resultant AST, which it then merges with the main AST. Thus the AST structure is also abstracted from the user, unlike Ruby and Groovy that allows explicit manipulation of the abstract syntax tree by the user. For details of Converge compile time meta-programming have a look at the Converge site.

Some languages like Nemerle and MetaLua allow dynamic extension of the language grammar through macros. Like Lisp in both of them, macros are not first class citizens, but help implement syntactic extensions in their own unique ways.

So long Haskell has been doing lots of DSL development based on pure embedding using powerful features like monadic interpreters, lazy evaluation and higher order function composition. But macros add yet another level of expressivity in language syntax, not possible through embedding alone. Are we seeing a new and invigorated effort towards implementing syntactic extensions to programming languages ? And does this have any relation to the recent interest and advancements in DSL based development ?

From http://debasishg.blogspot.com