Macro defining syntax proposal

[tcallred]

In response to the recent discussion on macros on Rhombus, I would like to share my own ideas on how we can make macro definitions more ergonomic and user friendly. The main idea is to remove incidental details so that the programmer can focus on essential parts of the macro definition.

Today, defining a macro looks something like this (from the docs):

fun
| factorial(0): 1
| factorial(n): n*factorial(n-1)

expr.macro '($a *! $tail ......):
  values('(factorial($a)), tail)

10*!

Defining a macro requires using expr.macro with a template denoted by '( ... ), and a block that returns values where the first argument is a template with the resulting expanded code and the second argument is matched syntax to be handled by something else. (You can also define a macro using def without the need for values).

In this post, I would like to propose a few changes to the syntax for the sake of discussion. To do that, I will use a toy-example macro that imitates python's list comprehensions.

(In today's macro system)

expr.macro '($exp for $a ~in $lst):
  values('(base.map(fun($a): $exp , $lst)), '())

(x*x) for x ~in [1, 2, 3] // -> [1, 4, 9]

Support for single return

When writing a macro that consumes all of its matched input and would return values( ..., '()), it would be nice to be able to simply return the template without the need to call values.

expr.macro '($exp for $a ~in $lst):
  '(base.map(fun($a): $exp , $lst))

This would implicitly produce values('(base.map(fun($a): $exp , $lst)), '()). This would make it similar to how def creates macros when given a template and a body with a resulting template. It also just makes it a little bit cleaner.

Templates created with back-ticks or single-quotes

Instead of using single-quote as a prefix operator paired with parentheses '(...) to create a template, we could use back-ticks or single-quotes as a new type of parentheses. This would require an extension to shrubbery but could make templates cleaner by removing extra parentheses.

My first idea involved using back-ticks, but after some discussion with @mflatt, it seemed like it would be better to use single quotes, specifically to avoid clashing with markdown. Below, I've included examples using both for the sake of discussion.

Using back-ticks:

expr.macro `$exp for $a ~in $lst`:
  `base.map(fun($a): $exp , $lst)`

Pros:

• Removing extra parentheses cleans it up by removing distraction from the content of the template
• Back-tics are typically associated with "code" and sends the message of something "different" from a string
• Relates to JavaScript's interpolated strings, helping people make the mental connection between macro templates and string interpolation.

Cons:

• Would require a major extension to shrubbery.
• Would conflict with the meaning of back-ticks in markdown documentation

Using single-quote

expr.macro '$exp for $a ~in $lst':
  'base.map(fun($a): $exp , $lst)'

Pros:

• Maintains the cleanliness just like back-tick
• Does not conflict with markdown syntax
• Would be relatively easy to extend shrubbery for considering that today single-quote is only read as an operator and could be changed act like parens
• Relates to Racket's '() forms

Cons:

• Would still need to change shrubbery
• Single quotes might be confusing for those used to using them for strings or chars
• Harder to distinguish from double-quote

Other examples with proposed ideas

expr.macro '$a *! $tail ... ':
  values( 'factorial($a)', tail )

expr.macro 'prefix_plus $a $b $c ...':
  values( a, '+ $b $c ...' )

expr.macro 'also_prefix_plus $e ...':
  match e
  | '$a $b $c ...':
      values( a, '+ $b $c ...' )
  | ~else:
      '"this is terrible error reporting"'

[sorawee]

expr.macro is for low-level macro. You can use expr.rule which already "supports for single return"

#lang rhombus

import:
  rhombus/macro open

fun
| factorial(0): 1
| factorial(n): n*factorial(n-1)

expr.rule '($a *!):
  '(factorial($a))

10*!

[mflatt]

I like the idea of having ' as a kind of parenthesis and using it for syntax quotes. That change would make things less noisy by removing the need for some parentheses, it helps avoids having some character combinations inconveniently treated as a single operator (such as '$ versus ' followed by $), and it removes the need for something like syntax_term to represent a term without parentheses. Also, balanced quotes are probably look more familiar to many. Although it would mean a new reserved character in shrubbery notation, it looks like a straightforward change, and having four kinds of parentheses instead of three kinds doesn't seem bad.

[sorawee]

Currently, we can do

'('(f(1)))

If we use ' as a parenthesis, how would we write this?

[mflatt]

One possibility is

'( 'f(1)' )'

You could read this as ( ' being an opener for a nested quote that's closed by ' ).

[samdphillips]

My concern with this proposal is that using a single token to represent the opening and closing of a syntax quote will make writing a macro that expands to something containing a syntax quote difficult to write/read.

[mflatt]

I think the difference compared to now would mostly be adding a ' at the end. That's because you almost always have to write ( after ' currently (for all but the simplest quoted terms). That ( with its closing ) would prevent a nested ' from matching outside the parentheses.

[gus-massa]

It's probably 50 year too late to change this, but I hate that most (all?) programming languages use the same character to start and end an string "Hello", and LaTeX use the same symbol to start and end math mode $x^2$.

(Preemptive reply: Yes, but nobody uses \(x^2\). Moreover, $$x^2$$ is even worse.)

[benknoble]

Not sure why this is relevant to the discussion being had here, but FWIW I have $ bound to insert \( and \) as appropriate (in vim), so some people do in fact use it.

[rocketnia]

This has been a pet peeve of mine too, as one of a few long-standing things I think could use fine-tuning in string syntaxes to let escape sequences be avoided in common cases. The string syntaxes I've pursued look more like at-expressions.

That said, I think the '...' syntax quotes look surprisingly conventional, even cozy. I like this change from a conventional syntax perspective, especially knowing that at-expressions and the Racket reader are right there as alternatives for more niche kinds of quoting concerns.

[mflatt]

I've pushed the change to use '...' for quoting syntax. The "demo.rhm" diff at the top of f216798 (you'll probably have to click to load the diff) illustrates the improvement.

As one example, the macro

defn.macro '(def_five $id):
  '(:
      def $id: 5
  )

becomes

defn.macro 'def_five $id':
  'def $id: 5'

Initially, I expected to push this change in a branch, so we could try out it. But it ended up being an involved change, and part of the reason things get better is that the change enabled and forced a generalization of syntax objects to gracefully handle single-term, single-group, and multi-group syntax objects. So, whatever syntax we settle on, I'm pretty sure the path runs through this version of the implementation.

The generalization to different kinds of syntax objects is why def_five above gets so much better: the def $id 5 part doesn't have to be packaged in a fake block. Multiple result groups work the way you'd expect:

defn.macro 'def_five_and_six $id5 $id6':
  'def $id5: 5
   def $id6: 6'

[rocketnia]

I think the $[def_form] notation looks fine, and macro-generating macros are rare enough that something like this might be more than enough for real programs, but I think it might not be ideal. (I'm going to bring up some more unsual design thoughts I've been having for Cene.)

Generally, it would be easy to write DSLs in most languages if code of the form lambda(list(local("x")), call(global("+"), list(local("x"), local("x")))) were tolerable to maintain. I think one of the main appeals of quoted s-expressions for DSL purposes is the way they allow variables to be written x rather than having all these delimiters. I think this is also why lots of programmers who otherwise prefer s-expressions still find ,x more readable than (unquote x).

If the language were to have no particular support, programmers could probably get by with something like $(quote_syntax($def_form)), essentially replicating the ,',def_form technique for escaping from multiple levels of quasiquotation. But unlike ,',def_form, something like $(quote_syntax($def_form)) is still more like (unquote x) than ,x in terms of clutter.

I think $[def_form] is better but still has some of that clutter remaining in the form of brackets.

If people consider it reasonable, I think it might be better to put all the clutter on the left side, such as $!>!def_form, $.>.def_form, $.def_form, $-def_form, or ${>}def_form. That allows the overall snippet to have a jagged silhouette, more like the silhouette a simple identifier would have had, making it potentially less distracting from the code the programmer is really trying to envision when they write it.

I listed a bunch of possibilities there. In the ones that use ! and {, I'm picking relatively rough-looking separators so they don't visually break up the "identifier" into pieces as crisply as a ( or | would. In the ones that use ., I'm attempting to appeal to the existing conventional use of . as a namespace qualification operator for identifiers.

And the point of > is to allow us to write sequences like $!>>>>!def_form when we want to escape through multiple stages of quotation at a time. (In $.def_form and $-def_form, I'm being a bit bolder and imagining the use of something a little terser like $....def_form or $----def_form.)

I think each > should have a corresponding < annotating the pair of quotation brackets that it escapes from. Instead of writing the opening bracket as ', we'd write it as $!<!' or '!<!$, or maybe the terser '-$ or $-' if it's matching up with something like $-def_form. (I'm using $ as part of the notation for annotated quotation brackets here so that it's possible for escaped brackets to resemble both unescaped brackets and other escape sequences.)

To keep this from being endlessly confusing, I think unannotated uses like ' and $foo should match up, and annotated uses like $!<!' and $!>!foo should match up, but they shouldn't pass through each other or interact in any way. Basically, they should be like (..) and [..], where a mismatched (..] is always treated as a mistake. And although $!>>>>!foo should interact with multiple $!<!' brackets, the $!<!' brackets it interacts with should have no unannotated ' brackets in between.

Overall, since majority of macros aren't macro-generating macros, this lets most code completely ignore the < annotations. The code that actually needs to manage multiple stages of quotation must start explicitly annotating every stage rather than using a potentially confusing mix of explicit and implicit interactions.

Actually, I think there's a deeper semantic purpose to keeping things explicit here: For some kinds of quotation, escaping from multiple layers of quotation might not have semantics quite as simple as $(quote_syntax($def_form)). It's possible some stages might use some slightly different kind of re-quotation there, like (quote-syntax def_form #:local) or (syntax/loc stx def_form), and some stages might perform sanitization or ascribe types or contracts to the things passing through. Making multiple-stage leaps more explicit using < and > should make it more obvious where in the code these other subtle choices can be written down and reviewed.

To make this all a bit more specific and tangible, this is what I'm imagining:

defn.macro 'def_a_and_b $def_form':
  $!<!｢
    defn.macro '$!>!def_form $id5 $id6':
      '
        def $id5: 5
        def $id6: 6
      '
  ｣

Besides the $!>! stuff, this example also incorporates some shallower stylistic tweaks:

• I've used ｢ and ｣ here for one of the pairs of quotation brackets. I think picking another Unicode character like this might be better than doubling up on the meaning of «. (Maybe Rhombus is already using ! too, in which case another character might be better there.)

• I've indented by a whole tab stop at a time. I often work with text editors like Notepad++ and Visual Studio Code that make it a pain to indent by variable numbers of spaces, and in terms of readability, I've always had to do a double take when I see two adjacent lines that differ by only one space of indentation, since that looks so close to being a mistake.

• I've used an indentation policy where if a pair of matching brackets spans multiple lines, then the lines it begins and ends on have the same amount of indentation. I think this tends to reflect how codebases in languages like C treat their curly brackets (though not always the way they treat their function call parentheses). In this case, besides looking more conventional, it avoids an asymmetrical phenomenon that would otherwise be caused by my whole-tab-stop-oriented indentation style, where opening brackets end up leading a much more wallflowery life than their corresponding closing brackets:

  defn.macro '$!>!def_form $id5 $id6':
    '
      def $id5: 5
      def $id6: 6'
  

[mflatt]

In trying out documentation updates for $[], I discovered that I had defined ... to also be sensitive to nesting depth — which seems natural and consistent with $, but runs into the same trouble as $ escape levels and doesn't have a natural place to add []. In Racket, meanwhile, ... is not sensitive to nesting depth; there's a totally different and special (and not popular) mechanism for quoting ... in templates. So, after trying out some alternatives, I now think that $[] may be trying too hard, while $('$') plus $('...') is not so bad and is at least simple and consistent.

@rocketnia's idea of a family of quoting an unquoting forms, like @|<<{ ... }>>| with escaping by |>>@, has worked well for reader-level quoting for at-notation. But for this to be used for patterns and templates, I think it would have to be lifted to a notion of parenthesis families and accompanying operators at the level of shrubbery "output" (i.e., the representation of a shrubbery term). that seems like a lot of extra complexity, too, and there would still be the ... problem.

[Lazerbeak12345]

I'm actually warming up to $('$') syntax. reasons are, primarily that

• Macros inside macros are very rare.
• It's simpler to parse
• It is making use of normal escaping, in a sense.

[rocketnia]

But for this to be used for patterns and templates, I think it would have to be lifted to a notion of parenthesis families and accompanying operators at the level of shrubbery "output" (i.e., the representation of a shrubbery term). that seems like a lot of extra complexity, too[...]

Yeah, that's where the complexity would go, and it would be a bunch of complexity for a niche purpose.

the ... problem

Hmm, that's also an escape sequence. Some options that you've probably already considered:

• $[foo] [...]
• [$[foo]] ...
• $[foo] ...[]
• $[foo] []...
• `📜$foo 📜...' (just picking some rare character to serve as a third escape sequence marker that interacts with the others)
• $$foo $... (based on @gus-massa's $$foo suggestion [edit: which I just noticed @Lazerbeak12345 suggested above])
• Redesign it to be written $..., and go with:
  • $[foo] $[...]
  • $$foo $$...

So, after trying out some alternatives, I now think that $[] may be trying too hard, while $('$') plus $('...') is not so bad and is at least simple and consistent.

Would these not give errors like "expected expression after $ in '$'" and "expected template term before ellipses in '...'"? Not that there's anything wrong with having a special case for something like this--all the other options seem to be special cases in their own ways--but I wonder if this has been taken into account.

[mflatt]

Would these not give errors like "expected expression after $ in '$'" and "expected template term before ellipses in '...'"?

No. As you say, a special case has been carved out for $ and ... by themselves in a template:

> '$'
'$'
> '...'
'...'
> '... x'
string:1:1: ...: misplaced repetition
  in: ...
 [,bt for context]

The following current behavior is is an accidental, though, and probably should be changed.

> 'x $'
'x $'

[gus-massa]

I like the general idea of rhombus of using indentation to define blocks, but I think that blocks of code in macros are weird enough to go back to matched parenthesis. It's slightly similar to strings, that really need a starting and an ending border.

I insist in using a different delimiter for each border, so nesting is easier. And my idea is to use $$to escape two levels instead of one, with the obvious generalization to $$$...

I'm not sure if this makes sense, but my proposal is

defn.macro «def_a_and_b $def_form»:
  «defn.macro «$$def_form $id5 $id6»:
     «def $id5: 5
      def $id6: 6»»