Over the years, Lisp programmers have discovered useful techniques for writing macros, and have identified pitfalls that must be avoided. This section discusses some of these techniques, and illustrates them with examples.
The most important thing to keep in mind as you learn to write macros is that you should first figure out what the macro form is supposed to expand into, and only then should you start to actually write the code of the macro. If you have a firm grasp of what the generated Lisp program is supposed to look like, from the start, you will find the macro much easier to write.
In general any macro that can be written as an inline function should be written as one, not as a macro, for several reasons:
See the section "Inline Functions". A macro can be an inline function only if it has the exact semantics of a function, rather than a special form.
One of the most common errors in writing macros is best illustrated by example. Suppose we wanted to write dolist as a macro that expanded into a do. The first step, as always, is to figure out what the expansion should look like. Let's pick a representative example form, and figure out what its expansion should be. Here is a typical dolist form.
(dolist (element (append a b))
(push element *big-list*)
(foo element 3))
We want to create a do form that does the thing that the above dolist form says to do. That is the basic goal of the macro: it must expand into code that does the same thing that the original code says to do, but it should be in terms of existing Lisp constructs. The do form might look like this:
(do ((list (append a b) (cdr list))
(element))
((null list))
(setq element (car list))
(push element *big-list*)
(foo element 3))
Now we could start writing the macro that would generate this code, and in general convert any dolist into a do, in an analogous way. However, there is a problem with the above scheme for expanding the dolist. The above expansion works fine. But what if the input form had been the following:
(dolist (list (append a b))
(push list *big-list*)
(foo list 3))
This is just like the form we saw above, except that the user happened to decide to name the looping variable list rather than element. The corresponding expansion would be:
(do ((list (append a b) (cdr list))
(list))
((null list))
(setq list (car list))
(push list *big-list*)
(foo list 3))
This does not work at all! In fact, this is not even a valid program, since it contains a do that uses the same variable in two different iteration clauses.
Here is another example that causes trouble:
(let ((list nil))
(dolist (element (append a b))
(push element list)
(foo list 3)))
If you work out the expansion of this form, you see that there are two variables named list, and that the user meant to refer to the outer one but the generated code for the push actually uses the inner one.
The problem here is an accidental name conflict. This can happen in any macro that has to create a new variable. If that variable ever appears in a context in which user code might access it, it might conflict with some other name that is in the user's program.
One way to avoid this problem is to choose a name that is very unlikely to be picked by the user, simply by choosing an unusual name. This will probably work, but it is inelegant since there is no guarantee that the user will not happen to choose the same name. The only sure way to avoid the name conflict is to use an uninterned symbol as the variable in the generated code. The function (make-symbol "foo") is useful for creating such symbols.
Here is the expansion of the original form, using an uninterned symbol created by (make-symbol "foo")
(do ((#:g0005 (append a b) (cdr #:g0005))
(element))
((null #:g0005))
(setq element (car #:g0005))
(push element *big-list*)
(foo element 3))
This is the right kind of thing to expand into. Now that we understand how the expansion works, we are ready to actually write the macro. Here it is:
(defmacro dolist ((var form) . body)
(let ((dummy (gensym)))
`(do ((,dummy ,form (cdr ,dummy))
(,var))
((null ,dummy))
(setq ,var (car ,dummy))
,@body)))
Many system macros do not use (make-symbol "foo") for the internal variables in their expansions. Instead they use symbols whose print names begin and end with a dot. This provides meaningful names for these variables when looking at the generated code and when looking at the state of a computation in the Debugger. However, this convention means that users should avoid naming variables this way.
prog-context Conflicts
A related problem occurs when you write a macro that expands into a prog (or a do, or something that expands into prog or do) behind the user's back.
Consider the error-restart special form; suppose we wanted to implement it as a macro that expands into a prog. If it expanded into a standard prog, then the following (contrived) Lisp program would not behave correctly:
(prog ()
(setq a 3)
(error-restart (error "Try again")
(cond ((> a 10)
(return 5))
((> a 4)
(fsignal 'lose "You lose."))))
(setq b 7))
The problem is that the return returns from the error-restart instead of the prog. The way to avoid this problem is to use a named prog whose name is t. The name t is special in that it is invisible to the return function. If we write error-restart as a macro that expands into a prog named t, then the return passes right through the error-restart form and returns from the prog, as it ought to. Don't use this mechanism if the looping construct needs a return from its own operation, since in that case the return will exit too far. You might want to use tagbody instead.
In general, when a macro expands into a prog or a do around the user's code, the prog or do should be named t so that return forms in the user code return to the right place, unless the macro is documented as generating a prog/do-like form that can be exited with return.
Sometimes a macro wants to do several different things when its expansion is evaluated. Another way to say this is that sometimes a macro wants to expand into several things, all of which should happen sequentially at run time (not macro-expand time). For example, suppose you wanted to implement defparameter as a macro. defparameter must do two things: declare the variable to be special, and set the variable to its initial value. To simplify the example, we implement a simplified defparameter that does only these two things, and does not have any options. What should a defparameter form expand into? What we would like is for an appearance of:
(defparameter a (+ 4 b))
in a file to be the same thing as the appearance of the following two forms:
(proclaim '(special a))
(setq a (+ 4 b))
However, because of the way that macros work, they expand into only one form, not two. So we need to have a defparameter form expand into one form that is just like having two forms in the file.
There is such a form. It looks like this:
(progn
(proclaim '(special a))
(setq a (+ 4 b)))
In interpreted Lisp, it is easy to see what happens here. This is a progn special form, and so all its subforms are evaluated, in turn. The proclaim form and the setq form are evaluated, and so each of them happens, in turn. So far, so good.
The interesting thing is the way this form is treated by the compiler. The compiler specially recognizes any progn form at top level in a file. When it sees such a form, it processes each of the subforms of the progn just as if that form had appeared at top level in the file. So the compiler behaves exactly as if it had encountered the proclaim form at top level, and then encountered the setq form at top level, even though neither of those forms was actually at top level (they were both inside the progn). This feature of the compiler is provided specifically for the benefit of macros that want to expand into several things.
Here is the macro definition:
(defmacro defparameter (variable init-form)
`(progn
(proclaim '(special ,variable))
(setq ,variable ,init-form)))
Here is another example of a form that wants to expand into several things. We will implement a special form called define-my-command, which is intended to be used in order to define commands in some interactive user subsystem. For each command, the define-my-command form provides two things: a function that executes the command, and a text string that contains the documentation for the command (in order to provide an online interactive documentation feature). This macro is a simplified version of a macro that is actually used in the Zwei editor. Suppose that in this subsystem, commands are always functions of no arguments, documentation strings are placed on the help property of the name of the command, and the names of all commands are put onto a list. A typical call to define-my-command would look like:
(define-my-command move-to-top
"This command moves you to the top."
(do ()
((at-the-top-p))
(move-up-one)))
This could expand into:
(progn
(defprop
move-to-top
"This command moves you to the top."
help)
(push 'move-to-top *command-name-list*)
(defun move-to-top ()
(do ()
((at-the-top-p))
(move-up-one)))
)
The define-my-command expands into three forms. The first one sets up the documentation string and the second one puts the command name onto the list of all command names. The third one is the defun that actually defines the function itself. Note that the defprop and push happen at load time (when the file is loaded); the function, of course, also gets defined at load time. For more discussion of the differences among compile time, load time, and eval time: See the function eval-when.
This technique makes Lisp a powerful language in which to implement your own language. When you write a large system in Lisp, frequently you can make things much more convenient and clear by using macros to extend Lisp into a customized language for your application. In the above example, we have created a little language extension: a new special form that defines commands for our system. It lets the writer of the system put documentation strings right next to the code that they document, so that the two can be updated and maintained together. The way that the Lisp environment works, with load-time evaluation able to build data structures, lets the documentation database and the list of commands be constructed automatically.
There is a particular kind of macro that is very useful for many applications. This is a macro that you place "around" some Lisp code, in order to make the evaluation of that code happen in some context. For a very simple example, we could define a macro called with-output-in-base, that executes the forms within its body with any output of numbers that is done defaulting to a specified base.
(defmacro with-output-in-base ((base-form) &body body)
`(let ((base ,base-form))
. ,body))
A typical use of this macro might look like:
(with-output-in-base (*default-base*)
(print x)
(print y))
that would expand into:
(let ((base *default-base*))
(print x)
(print y))
This example is too trivial to be very useful; it is intended to demonstrate some stylistic issues. Some special forms are similar to this macro. See the function with-open-file. See the function with-input-from-string.
The really interesting thing, of course, is that you can define your own such special forms for your own specialized applications. One very powerful application of this technique is used in a system that manipulates and solves the Rubik's cube puzzle. The system heavily uses a special form called with-front-and-top, whose meaning is "evaluate this code in a context in which this specified face of the cube is considered the front face, and this other specified face is considered the top face".
The first thing to keep in mind when you write this sort of macro is that you can make your macro much clearer to people who might read your program if you conform to a set of loose standards of syntactic style. By convention, the names of such special forms start with "with-". This seems to be a clear way of expressing the concept that we are setting up a context; the meaning of the special form is "do this with the following things true". Another convention is that any "parameters" to the special form should appear in a list that is the first subform of the special form, and that the rest of the subforms should make up a body of forms that are evaluated sequentially with the last one returned. All of the examples cited above work this way. In our with-output-in-base example, there was one parameter (the base), which appears as the first (and only) element of a list that is the first subform of the special form. The extra level of parentheses in the printed representation serves to separate the "parameter" forms from the "body" forms so that it is textually apparent which is which; it also provides a convenient way to provide default parameters (a good example is the with-input-from-string special form, which takes two required and two optional "parameters"). Another convention/technique is to use the &body keyword in the defmacro to tell the editor how to correctly indent the special form. See the macro defmacro.
The other thing to remember is that control can leave the special form either by the last form's returning, or by a nonlocal exit (that is, something doing a throw). You should write the special form in such a way that everything is cleaned up appropriately no matter which way control exits. In our with-output-in-base example, there is no problem, because nonlocal exits undo lambda-bindings. However, in even slightly more complicated cases, an unwind-protect form is needed: the macro must expand into an unwind-protect that surrounds the body, with "cleanup" forms that undo the context-settingup that the macro did. For example, using-resource is implemented as a macro that does an allocate-resource and then performs the body inside of an unwind-protect that has a deallocate-resource in its "cleanup" forms. This way the allocated resource item is deallocated whenever control leaves the using-resource special form.
In any macro, you should always pay attention to the problem of multiple or out-of-order evaluation of user subforms. Here is an example of a macro with such a problem. This macro defines a special form with two subforms. The first is a reference, and the second is a form. The special form is defined to create a cons whose car and cdr are both the value of the second subform, and then to set the reference to be that cons. Here is a possible definition:
(defmacro test (reference form)
`(setf ,reference (cons ,form ,form)))
Simple cases work all right:
(test foo 3) =>
(setf foo (cons 3 3))
But a more complex example, in which the subform has side effects, can produce surprising results:
(test foo (setq x (1+ x))) =>
(setf foo (cons (setq x (1+ x))
(setq x (1+ x))))
The resulting code evaluates the setq form twice, and so x is increased by two instead of by one. A better definition of test that avoids this problem is:
(defmacro test (reference form)
(let ((value (gensym)))
`(let ((,value ,form))
(setf ,reference (cons ,value ,value)))))
With this definition, the expansion works as follows:
(test foo (setq x (1+ x))) =>
(let ((#:g0005 (setq x (1+ x))))
(setf foo (cons #:g0005 #:g0005)))
In general, when you define a new special form that has some forms as its subforms, you have to be careful about when those forms get evaluated. If you are not careful, they can get evaluated more than once, or in an unexpected order, and this can be semantically significant if the forms have side effects. There is nothing fundamentally wrong with multiple or out-of-order evaluation if that is really what you want and if it is what you document your special form to do. However, it is very common for special forms to simply behave like functions, and when they are doing things like what functions do, it is natural to expect them to be function-like in the evaluation of their subforms. Function forms have their subforms evaluated, each only once, in left-to-right order, and special forms that are similar to function forms should try to work that way too for clarity and consistency.
The macro once-only makes it easier for you to follow the principle explained above. It is most easily explained by example. The way you would write test using once-only is as follows:
(defmacro test (reference form)
(once-only (form)
`(setf ,reference (cons ,form ,form))))
This defines test in such a way that the form is evaluated only once, and references to form inside the macro body refer to that value. once-only automatically introduces a lambda-binding of a generated symbol to hold the value of the form. Actually, it is more clever than that; it avoids introducing the lambda-binding for forms whose evaluation is trivial and may be repeated without harm or cost, such as numbers, symbols, and quoted structure. This is just an optimization that helps produce more efficient code.
The once-only macro makes it easier to follow the principle, but it does not completely nor automatically solve the problems of multiple and out-of-order evaluation. It is just a tool that can solve some of the problems some of the time; it is not a panacea.
The following description attempts to explain what once-only does, but it is much easier to use once-only by imitating the example above than by trying to understand once-only's rather tricky definition.
A once-only form looks like this:
(once-only (variable-name &environment environment)
form1
form2
...)
variable-name is a list of variables. once-only is usually used in macros where the variables are Lisp forms. &environment should be followed by a single variable that is bound to an environment representing the lexical environment in which the macro is to be interpreted. Typically this comes from the &environment parameter of a macro. The forms are a Lisp program that presumably uses the values of the variables to construct a new form to be the value of the macro. When a call to the macro that includes the once-only form is macroexpanded, the form produced by that expansion will be evaluated.
The macro that includes the once-only form will be macroexpanded. The form produced by that expansion is then evaluated. In the process, the values of each of the variables in variable-name are first inspected. These variables should be bound to subforms, that probably originated as arguments to the defmacro or similar form, and will be incorporated in the macro expansion, possibly in more than one place.
Each variable is then rebound either to its current value, if the current value is a trivial form, or to a generated symbol. Next, once-only evaluates the forms, in this new binding environment, and when they have been evaluated it undoes the bindings. The result of the evaluation of the last form is presumed to be a Lisp form, typically the expansion of a macro. If all of the variables had been bound to trivial forms, then once-only just returns that result. Otherwise, once-only returns the result wrapped in a lambda-combination that binds the generated symbols to the result of evaluating the respective nontrivial forms.
The effect is that the program produced by evaluating the once-only form is coded in such a way that it only evaluates each of the forms that are the values of variables in variable-name once, unless evaluation of the form has no side effects. At the same time, no unnecessary lambda-binding appears in the program. The body of the once-only is not cluttered up with extraneous code to decide whether or not to introduce lambda-binding in the program it constructs.
Note well: once-only can be used only with an &environment keyword argument. If this argument is not present, a compiler warning will result.
For more information about using once-only with &environment: See the lambda list keyword &environment. Also, refer to the definitions of the macro defining forms: defmacro, macrolet, and defmacro-in-flavor.
(defmacro double (x &environment env)
(once-only (x &environment env)
`(+ ,x ,x)))
=> DOUBLE
(double 5)
=> (+ 5 5)
(double var)
=> (+ VAR VAR)
(double (compute-value var))
=> (LET ((#:ONCE-ONLY-X-3553 (COMPUTE-VALUE VAR)))
(+ #:ONCE-ONLY-X-3553 #:ONCE-ONLY-X-3553))
Note that in the first three examples, when the argument is simple, it is duplicated. In the last example, when the argument is complicated and the duplication could cause a problem, it is not duplicated.
For information about avoiding problems with evaluation: See the section "Avoiding Multiple and Out-of-Order Evaluation".
once-only evaluates its subforms in the order they are presented. If it finds any form which is non-trivial, it rebinds the earlier variables to temporaries, and evaluates them first. In the following example, the order of evaluation is x, then y, even though the y appears before the x in the body of the once-only:
(defmacro my-progn (x y &environment env)
(once-only (x y &environment env)
;; We willfully try to make it evaluate in the wrong order.
`(progn ,y ,x))) => MY-PROGN
;;Macro expansion shows code that would be produced by the
;; once-only form in the macro.
(my-progn (print x) (setq x 'foo)) =>
(LET ((#:ONCE-ONLY-X-7614 (PRINT X)))
(PROGN (VALUES (SETQ X 'FOO)) #:ONCE-ONLY-X-7614))
In the next example, once-only evaluates y, then x, because y appears before x in once-only's variable list. In actuality, this style is an example of poor programming practice as it is confusing. Always list variables in the order in which the forms they are bound to appear in the source that produced them. In a macro, this is normally the order they appear in the macro's argument list.
(defmacro backward-progn (x y &environment env)
(once-only (y x &environment env)
;; We willfully try to make it evaluate in the wrong order.
;; But this time we tell once-only to evaluate y before x.
`(progn ,y ,x))) => BACKWARD-PROGN
(backward-progn (print x) (setq x 'foo)) => FOO
FOO
(PROGN (VALUES (SETQ X 'FOO)) (VALUES (PRINT X))) => FOO
FOO
Caution! A number of Zetalisp macros, zl:setf for example, fail to follow this convention. Unexpected multiple evaluation and out-of-order evaluation can occur with them. This was done for the sake of efficiency and is prominently mentioned in the documentation of these macros. It would be best not to compromise the semantic simplicity of your own macros in this way. (cl:setf and related macros follow the convention correctly.)
A useful technique for building language extensions is to define programming constructs that employ two special forms, one of which is used inside the body of the other. Here is a simple example. There are two special forms; the outer one is called with-collection, and the inner one is called collect. collect takes one subform, which it evaluates; with-collection just has a body, whose forms it evaluates sequentially. with-collection returns a list of all of the values that were given to collect during the evaluation of the with-collection's body. For example:
(with-collection
(dotimes (i 5)
(collect i)))
=> (1 2 3 4 5)
Remembering the first piece of advice we gave about macros, the next thing to do is to figure out what the expansion looks like. Here is how the above example could expand:
(let ((#:g0005 nil))
(dotimes (i 5)
(push i #:g0005))
(nreverse #:g0005))
Now, how do we write the definition of the macros? with-collection is pretty easy:
(defmacro with-collection (&body body)
(let ((var (gensym)))
`(let ((,var nil))
,@body
(nreverse ,var))))
The hard part is writing collect. Let's try it:
(defmacro collect (argument)
`(push ,argument ,var))
Note that something unusual is going on here: collect is using the variable var freely. It is depending on the binding that takes place in the body of with-collection to get access to the value of var. Unfortunately, that binding took place when with-collection got expanded; with-collection's expander function bound var, and it got unbound when the expander function was done. By the time the collect form gets expanded, var has long since been unbound. The macro definitions above do not work. Somehow the expander function of with-collection has to communicate with the expander function of collect to pass over the generated symbol.
The only way for with-collection to convey information to the expander function of collect is for it to expand into something that passes that information. What we can do is to define a special variable (which we will call *collect-variable*), and have with-collection expand into a form that binds this variable to the name of the variable that the collect should use. Now, consider how this works in the interpreter. The evaluator first sees the with-collection form, and calls in the expander function to expand it. The expander function creates the expansion, and returns to the evaluator, which then evaluates the expansion. The expansion includes in it a let form to bind *collect-variable* to the generated symbol. When the evaluator sees this let form during the evaluation of the expansion of the with-collection form, it sets up the binding and recursively evaluates the body of the let. Now, during the evaluation of the body of the let, our special variable is bound, and if the expander function of collect gets run, it is able to see the value of *collect-variable* and incorporate the generated symbol into its own expansion.
Writing the macros this way is not quite right. It works fine interpreted, but the problem is that it does not work when we try to compile Lisp code that uses these special forms. When code is being compiled, there is no interpreter to do the binding in our new let form; macro expansion is done at compile time, but generated code is not run until the results of the compilation are loaded and run. The way to fix our definitions is to use compiler-let instead of let. compiler-let is a special form that exists specifically to do the sort of thing we are trying to do here. compiler-let is identical to let as far as the interpreter is concerned, so changing our let to a compiler-let does not affect the behavior in the interpreter; it continues to work. When the compiler encounters a compiler-let, however, it actually performs the bindings that the compiler-let specifies, and proceeds to compile the body of the compiler-let with all of those bindings in effect. In other words, it acts as the interpreter would.
Here is the right way to write these macros:
(defvar *collect-variable*)
(defmacro with-collection (&body body)
(let ((var (gensym)))
`(let ((,var nil))
(compiler-let ((*collect-variable* ',var))
. ,body)
(nreverse ,var))))
(defmacro collect (argument)
`(push ,argument ,*collect-variable*))
A better way to write this type of macro is to use macrolet to create a definition of collect local to the body of with-collection. Example:
(defmacro with-collection (&body body)
(let ((var (gensym)))
`(let ((,var nil))
(macrolet ((collect (argument)
`(push ,argument ,',var)))
. ,body)
(nreverse ,var))))
;To make COLLECT known to editing tools, and to get a better error
;message if it is used in the wrong place, we define a global
;definition that will be shadowed by the MACROLET. The error
;message for misuse of COLLECT comes out at both compile-time
;and run-time.
(defmacro collect (argument)
(compiler:warn () "~S used outside of ~S"
'collect 'with-collection)
`(ferror "~S used outside of ~S"
`(collect ,,argument) 'with-collection))
The technique of defining functions to be used during macro expansion deserves explicit mention. A macro-expander function is a Lisp program like any other Lisp program, and it can benefit in all the usual ways by being broken down into a collection of functions that do various parts of its work. Usually macro-expander functions are pretty simple Lisp programs that take things apart and put them together slightly differently, but some macros are quite complex and do a lot of work. Several features of Symbolics Common Lisp, including flavors, loop, and defstruct, are implemented using very complex macros, which, like any complex, well-written Lisp program, are broken down into modular functions. You should keep this in mind if you ever invent an advanced language extension or ever find yourself writing a five-page expander function.
A particular thing to note is that any functions used by macroexpander functions must be available at compile time. You can make a function available at compile time by surrounding its defining form with (eval-when (compile load eval) ...). Doing this means that at compile time the definition of the function is interpreted, not compiled, and thus runs more slowly. Another approach is to separate macro definitions and the functions they call during expansion into a separate file, often called a "defs" (definitions) file. This file defines all the macros but does not use any of them. It can be separately compiled and loaded up before compiling the main part of the program, which uses the macros. The system facility helps keep these various files straight, compiling and loading things in the right order. See the section "System Construction Tool".
This special form goes into a loop in which it reads forms and sequentially expands them, printing out the result of each expansion (using the grinder to improve readability). See the section "Functions for Formatting Lisp Code". It terminates when you press the END key. If you type in a form that is not a macro form, there are no expansions and so it does not type anything out, but just prompts you for another form. This allows you to see what your macros are expanding into, without actually evaluating the result of the expansion.
For example:
(mexp)
Type End to stop expanding forms
Macro form -> (loop named t until nil return 5)
(LOOP NAMED T UNTIL NI RETURN 5) ->
(PROG T NIL
SI:NEXT-LOOP AND NIL
(GO SI:END-LOOP))
(RETURN 5)
(GO SI:NEXT-LOOP)
SI:END-LOOP)
Macro form -> (defparameter foo bar) ->
(PROGN (EVAL-WHEN (COMPILE)
(COMPILER:SPECIAL-2 'FOO))
(EVAL-WHEN (LOAD EVAL)
(SI:DEFCONST-1 FOO BAR NIL)))
See the section "Expanding Lisp Expressions in Zmacs". That section describes two editor commands that allow you to expand macros c-sh-M and m-sh-M. There is also the Command Processor command, Show Expanded Lisp Code. See the document Genera User's Guide.