Overview

A macro is an extension to the core language that can be defined by the user, by the implementation, or as part of the Dylan language specification. Much of the grammatical structure of Dylan is built with macros. A macro defines the meaning of one construct in terms of another construct. The compiler substitutes the new construct for the original. The purpose of macros is to allow programmers to extend the Dylan language, for example by creating new control structures or new definitions. Unlike C, Dylan does not intend macros to be used to optimize code by inlining. Other parts of the language, such as sealing and define constant, address that need.

Throughout this chapter, italic font and small caps are used to indicate references to the formal grammar given in Appendix A, BNF.

Compilation and Macro Processing

Compilation consists of six conceptual phases:

Parsing a stream of characters into tokens, according to the lexical grammar in Appendix A, BNF.
Parsing a stream of tokens into a program, according to the phrase grammar in Appendix A, BNF.
Macro expansion, which translates the program to a core language.
Definition processing, which recognizes special and built-in definitions and builds a compile-time model of the static structure of the program.
Optimization, which rewrites the program for improved performance.
Code generation, which translates the program to executable form.

Portions of a program can be macro calls. Macro expansion replaces a macro call with another construct, which can itself be a macro call or contain macro calls. This expansion process repeats until there are no macro calls remaining in the program, thus macros have no space or speed cost at run time. Of course, expanding macros affects the speed and space cost of compilation.

A macro definition describes both the syntax of a macro call and the process for creating a new construct to replace the macro call. Typically the new construct contains portions of the old one, which can be regarded as arguments to the macro. A macro definition consists of a sequence of rewrite rules. The left-hand side of each rule is a pattern that matches a macro call. The right-hand side is a template for the expansion of a matching call. Pattern variables appearing in the left-hand side act as names for macro arguments. Pattern variables appearing in the right-hand side substitute arguments into the expansion. Macro arguments can be constrained to match specified elements of the Dylan grammar. Auxiliary rule sets enhance the rewrite rule notation with named subrules.

Some implementations and a future version of the Dylan language specification might allow macro expansions to be produced by compile-time computation using the full Dylan language and an object-oriented representation for programs. Such a procedural macro facility is not part of Dylan at this time.

The input to, and output from, macro expansion is a fragment, which is a sequence of elementary fragments. An elementary fragment is one of the following:

A token: the output of the lexical grammar. The bracket tokens (, ), [, ], {, }, #(, and #[ are not allowed. Core reserved words (except otherwise), begin-words, and function-words are not allowed unless quoted with backslash.
A bracketed fragment: balanced brackets ( (), [], or {} ) enclosing a fragment.
A macro call fragment: a macro call.
A parsed fragment: a single unit that is not decomposable into its component tokens. It has been fully parsed by the phrase grammar. A parsed fragment is either a function call, a list constant, a vector constant, a definition, or a local declaration.

The second and third phases of compilation (parsing and macro expansion) are interleaved, not sequential. The parsing phase of the compiler parses a macro call just enough to find its end. See definition-macro-call, statement, function-macro-call, body-fragment, list-fragment, and basic-fragment in Appendix A, BNF. This process of parsing a macro call also parses any macro calls nested inside it. The result is a macro call fragment.

This loose grammar for macro calls gives users a lot of flexibility to choose the grammar that their macros will accept. For example, the grammar of macro calls doesn't care whether a bracketed fragment will be interpreted as an argument list, a parameter list, a set of for clauses, or a module import list.

The compiler delays computing the expansion of a macro call fragment until it is needed. If an argument to a macro is a macro call, the outer macro call is always expanded first. When the compiler computes the expansion of a macro call fragment, it obeys the macro's definition. Constraints on pattern variables can cause reparsing of portions of the macro call.

A constituent, operand, or leaf that is a macro call expands the macro during or before the definition processing and optimization phases. The compiler brackets the expansion in begin … end, using the standard binding of begin in the Dylan module, and then reparses it as a statement. This reparsing may discover more macro calls. A parse error while reparsing a macro expansion could indicate an invalid macro definition or an incorrect call to the macro that was not detected during pattern matching. Once the cycle of macro expansion and reparsing has been completed, no tokens, bracketed fragments, or macro call fragments remain and the entire source record has become one parsed fragment.

This begin … end bracketing ensures that the expansion of a macro call will not be broken apart by operator precedence rules when the macro call is a subexpression. Similarly, it ensures that the scopes of local declarations introduced by a macro will not extend outside that macro expansion when the macro call is a statement in a body.

The fragment produced by parsing a macro call, which is the input to macro expansion, is as follows:

Local declarations and special definitions are parsed fragments.
Calls to macros are macro call fragments.
List constants and vector constants are parsed fragments.
Anything in brackets is a bracketed fragment.
If the macro call was not the result of macro expansion, everything else is represented as sequences of tokens. There are a few restrictions on the tokens, for example semicolons must appear in certain places and bare brackets cannot appear; for details see the definition of body-fragment and list-fragment in Appendix A, BNF.
In a macro call that is the result of macro expansion, additional items can be parsed fragments, due to pattern-variable substitution.
Many built-in macros expand into implementation-specific parsed fragments.

The fragment produced by parsing an expression is as follows:

An expression consisting of a single token returns a one-token fragment. This will be a variable-name, noncollection literal, or symbol.
An expression consisting of just a string literal returns a one-token fragment. If the string literal consists of multiple string tokens, they are concatenated into a single string token.
An expression consisting of just a list constant or a vector constant returns a list constant or vector constant fragment.
An expression consisting of just a statement or function-macro-call returns a macro call fragment.
An operator call, slot reference, or element reference that calls a function macro returns a macro call fragment.
A function call, operator call, slot reference, or element reference that calls something other than a function macro returns a function call fragment.
Enclosing an expression in parentheses does not change how it parses.

The term parsed expression fragment refers to any of the above.

The parser recognizes parsed fragments as well as raw tokens. The nonterminals definition and local-declaration in the phrase grammar accept parsed fragments of the same kind. The nonterminal operand accepts parsed function call fragments and macro call fragments. The nonterminal literal accepts list constant and vector constant fragments. The nonterminal simple-fragment accepts parsed function call fragments and macro call fragments. The nonterminal macro accepts macro call fragments. The parser expands bracketed fragments into their constituent tokens before parsing them.