[RFC] Named macro capture groups

[tgross35]

This RFC proposes optional names for repetition groups in macros:

macro_rules! foo {
    ( $group1( $a:ident ),+ ) => {
        $group1( println!("{}", $a); )+
    }
}

Rendered

Small Pre-RFC: https://internals.rust-lang.org/t/pre-rfc-named-capture-groups-for-macros/20883

[RalfJung]

Aren't there still some subtle questions around nesting one has to worry about? Specifically, the RFC doesn't seem to say anything about nested cases like ( $group1:( $a:ident [ $group2:( $b:ident ),+ ] ),+ ). Presumably I will only be allowed to use $group2 inside a $group1(...) repetition? But there can be arbitrarily many other repetitions "in between" -- group2 will always refer to the repetition instance in the "current" group1?

[davidhewitt]

Although I cannot remember the exact use-case, I remember wanting something like this to name an optional trailing comma so that I could forward it on to another macro as part of the expansion. 👍

[tgross35]

Aren't there still some subtle questions around nesting one has to worry about? Specifically, the RFC doesn't seem to say anything about nested cases like ( $group1:( $a:ident [ $group2:( $b:ident ),+ ] ),+ ). Presumably I will only be allowed to use $group2 inside a $group1(...) repetition? But there can be arbitrarily many other repetitions "in between" -- group2 will always refer to the repetition instance in the "current" group1?

Great point, the scope part is tricky - I think what you said is correct. It is probably best if I just add an appendix of examples that do and don't work with reasoning, especially given your comments on Zulip about the existing macro mental model not being very clear (I completely agree).

[RalfJung]

A bunch of examples would be great, but they are not substitute for also describing the general rules that they all follow from.

[kennytm]

when you write this I thought you are defining a metavariable $group1 which matches a comma-separated list of $a:ident, and using $group1 will expand to foo,bar,baz,etc.

but no the $group1 is just a label 🤔

the rationale mentioned using : is to be "to be more consistent with existing fragment specifiers" but IMO similarity with fragment specifiers is exactly why this RFC should not use the current syntax.

i prefer getting rid of : (my main source of confusion) and use a label syntax like

macro_rules! make_functions {
    ( 
        names: [ $'names $($name:ident),+ ],
//               ^~~~~~~~~
        $'greetings $( greeting: $greeting:literal, )?
    ) => {
        $'names $(
            fn $name() {
                println!("function {} called", stringify!($name));
                $'greetings $(println!("{}", $greeting) )?
            }
        )+
    }
}


macro_rules! innermost1 {
    ( $'outer_rep $($a:ident: $'inner_rep $($b:literal),* );+ ) => {
        [$'outer_rep $( $'inner_rep $( ${index($'outer_rep)}, )* )+]
    };
}

[petrochenkov]

Metavariables matching some parts of the input are an interesting idea in general.
If we also introduce "match exactly once repetitions", then we'll also be able to capture just arbitrary token streams.
It would be nice to also be able to bind a name to "repeated exactly one" groups aka just artibtrary substreams in the input.

// macro LHS, capture "exactly once" repetition
$my_tokens:(a b $var c d)①

// macro RHS, reemit `a b value_for_var c d`
$my_tokens

[tgross35]

Thanks for the ideas - I updated the default syntax to be $group1(...) (no colon) and added the label syntax as a possibility.

It would be nice to also be able to bind a name to "repeated exactly one" groups aka just artibtrary substreams in the input.

Is there a standard kleene for a single capture, or could we get by with no kleene at all? I could probably include a proposal for single captures in this RFC, to be accepted as part of it or rejected separately.

[RalfJung]

In regexes, a "capture exactly once" group is just (...).

[tgross35]

After writing some more examples, it seems like "capture once" and "emit the entire capture" would be pretty useful, so I just included them as part of the proposal. Especially for code that just needs to be validated and passed elsewhere, or matching and reemitting optional keywords/tts like mut, & or pub(crate) (just allowing capture groups that don't contain metavars is enough for that, but $pub_crate in the expansion is nicer to read than $pub_crate(pub(crate))).

[petrochenkov]

One advantage of the reemitting is that it will use spans from the input, instead of spans from the macro body.

One example I recently observed in rust-lang/rust#119412 is the mir! macro in standard library.
It will match on something like let $expr = $expr; and then reemit $expr = $expr; as an assignment expression.
If = is emitted manually from the macro it will get the macro span and may mess up the combined span of the assignment expression.
If = is taken from the input, then the expression will get a precise span from the input, which is good for any kind of DSLs.

[c410-f3r]

cc @markbt

[nikomatsakis]

Skimmed this: I feel positive on this direction.

[joshtriplett]

I love this specific point, and have wanted this in the past.

[joshtriplett]

One potential source of ambiguity:

How would you express "I want to expand the entire named group, followed by some tokens in parentheses"?

I have a suggestion for how to solve that: ${$group}(tokens).

[joshtriplett]

🎉 This is a great sign that this design is a good idea.

[joshtriplett]

@tgross35 I love this approach. This seems cleaner, nicely orthogonal, and would fit very well with macro fragment fields.

[joshtriplett]

I can see the argument for either $group:(...) or $group(...). The former seems nicely analogous to existing macro fragments ($x:ident $g:(...) defines two captures with their types), while the latter has less syntax, still parses unambiguously, and has the advantage that the usage of a group matches the input syntax.

[steffahn]

Syntax like e.g. $identifier( /* … */ ), is already valid and useful in macro expansions.

macro_rules! call_many_fuctions {
    ([$($($function_name:ident),+$(,)?)?]($arg:expr)) => {
        [
            $($(
                $function_name($arg),
            )+)?
        ]
    }
}
fn foo(x: i32) -> i32 { x }
fn bar(x: i32) -> i32 { x + 1 }
fn baz(x: i32) -> i32 { x * x }
fn qux(x: i32) -> i32 { x / 2 }

fn main() {
    println!("{:?}", call_many_fuctions!([foo, bar, baz, qux](100)));
}

(this concrete example doesn’t follow it up with a *, + or ?, but these aren’t forbidden either, of course)

If this RFC proposes to make the “parsing” of the macro RHS depend on the “type” of the fragment, I think that’s a clear possible disadvantage.

Especially with the (IMO very useful) idea that $group on its own can already stand in for an expansion of the whole group’s contents, so expanding some identifier $function_name – or expression, pattern, etc… – doesn’t work different from expanding some group $group – it seems desirable (to me) if $function_name($arg), *etc… doesn’t have a severely different meaning from $group($arg), *etc… either.

[kennytm]

the design that $group does not expand is primarily why i suggested in #3649 (comment) that it should use a separate syntax $'group.

[tgross35]

These are great points. I wish we could use the same syntax for metavars and groups, but the $'group syntax does get us out of the problems here, so I'll use that for the implementation and leave an unresolved question.

[bjorn3]

What if you want to use () after the group in the expansion? That would be interpreted as replacing the contents of the capture, right? Maybe it could be written as $...group or something like that?

[PoignardAzur]

As mentioned by other reviewers, the expansion-side macro is already valid.

I'd like to suggest an alternative:

macro_rules! foo {
    ( $group1( $a:ident ),+ ) => {

        ${for a in group1} (
            println!("{}", $a);
        )
    }
}

This alternative is more verbose, but it's also much more explicit about what's going on.

[PoignardAzur]

Or for something closer to existing syntax:

        ${for(group1)} (
            println!("{}", $a);
        )+

[PoignardAzur]

Something I like about everything else in the RFC is that capture groups are mostly distinct from their contents, and mostly serve as explicit scope. Being able to do $capture_group kind of loses that.

I think you should use an explicit syntax for emitting capture groups, maybe a metavariable like ${paste(capture_group)}:

macro_rules! rename_fn {
    (
        $newname:ident;
        $pub(pub)? fn $oldname:ident( $args( $mut(mut)? $arg:ident: $ty:ty )* );
    ) => {
        $pub fn $newname( ${paste(args)} );
    }
}

[chrisbouchard]

If exactly-once groups don't have a kleene operator to terminate the group matcher, how would a macro author define a rule that matches an exactly-once group followed by one of the tokens *, +, or ??

For example, in the following (admittedly kind of silly) example, I want to specify a multiplication-like syntax for my macro. But I think the * would be attached to the group instead, silently turning it into a zero-or-more group.

macro_rules! example1 {
    ( $g1($a:ident[$b:expr]) * $g2($c:ident[$d:expr]) ) => { $g1 $g2 }
}

example1!(x[1] * y[2]);  // ERROR: Something something unexpected `*`...

Further, a kleene-looking token could “reach back” across an intermediate token, interpreting it as a separator. I think this would be extra surprising, because an author may edit a macro definition and unintentionally change the the nature of the group at a distance.

For example, the following example would specify an exactly-once group.

macro_rules! example2 {
    ( $g($a:ident $b:ident) x y + ) => { $g }
}

example2!(foo bar x y +);  // Ok

But if I later edited it to remove the y token, it would silently become a one-or-more group with x as the separator.

macro_rules! example2 {
    ( $g($a:ident $b:ident) x + ) => { $g }
}

example2!(foo bar x +);  // ERROR: Something about `+` is not an ident and/or unexpected end of input...

For this reason, I don't think it would be a satisfying solution to just add an escape syntax for *, +, and ?. It would work, but it would still lead to gotchas like this where the macro author would have to

• Notice that their macro has broken, since the updated rule is still valid even if it doesn't match what they meant, and simple invocations may still work;
• Then reason out what went wrong from possibly-unhelpful error messages, before eventually slapping their head and adding an escape.

I don't think the compiler could help much here, because again these are all valid matchers. The compiler wouldn't know the author wanted an exactly-once group.

I think it would be better to introduce a new “kleene” operator for this new group type. (Scare quotes since Kleene didn't actually define a regular language operator like that, but he wasn't concerned with saving sub-matches anyway.) There isn't any symbol that I love for the job—maybe ! or =?

[chrisbouchard]

I do think there would be some backwards compatibility concerns with my suggestion if we allowed unnamed group matchers to use the new exactly-once operator. For example, consider the following macro definition.

macro_rules! example {
    ( $( $a:ident )!* ) => { $( $a ) }
}

Today, the rule matches zero or more idents with ! as the separator. If we made ! the new exactly-once operator, then I think this rule would instead match exactly one ident followed by *.

Maybe we only allow exactly-once groups with the new named syntax? (That might actually be a good idea regardless what syntax we use, since exactly-once groups are kind of useless without named groups.) But then would it be too surprising if !* meant different things for named and unnamed groups?

[chrisbouchard]

I realized my example in the previous comment wasn't quite right, because the group expansion in the transcriber needs an operator too. My example should have been written as follows, with a * in the transcriber.

macro_rules! example {
    ( $( $a:ident )!* ) => { $( $a )* }
}

Then I think expanding an exactly-once group using a zero-or-more operator should be an error. Allowing a new exactly-once operator with unnamed groups would still a backwards-compatibility concern, but it wouldn't be a silent backwards compatibility concern. (Nope. See below.)

---

I'm actually a bit unclear here, though. The reference for Macros by Example says the following about repetitions:

1. A metavariable must appear in exactly the same number, kind, and nesting order of repetitions in the transcriber as it did in the matcher. So for the matcher $( $i:ident ),*, the transcribers => { $i }, => { $( $( $i)* )* }, and => { $( $i )+ } are all illegal, …

— https://doc.rust-lang.org/reference/macros-by-example.html#r-macro.decl.repetition.fragment

However, it doesn't seem like this is currently enforced. (Playground example mixing * with +, and another mixing * with ?.) I don't know whether that's just a bug, or whether the reference is out of date.

Edit: Bleh, it's not a bug, but it's not exactly allowed either. It was reported in (at least) rust-lang/rust#103256 and the answer is there's an allow-by-default lint meta_variable_misuse that would catch it, but they don't want to make it warn- or deny-by-default because there are too many false positives.

---

I guess this is all kind of moot if we just don't allow exactly-once unnamed groups. And maybe for named groups, at least, things will be unambiguous enough to let us enforce that Kleene operators must match between matcher and transcriber.

[chrisbouchard]

I commented elsewhere about concerns with the proposed syntax for exactly-once groups. But with the current proposal, I think the grammar node would need the trailing MacroRepSep-MacroRepOp sequence to be optional.

Suggested change

	> `$` ( IDENTIFIER_OR_KEYWORD except crate | RAW_IDENTIFIER | _ )<sup>?</sup> ( _MacroMatch<sup>+</sup>_ ) _MacroRepSep_<sup>?</sup> _MacroRepOp_
	> `$` ( IDENTIFIER_OR_KEYWORD except crate | RAW_IDENTIFIER | _ )<sup>?</sup> `(` _MacroMatch<sup>+</sup>_ `)` ( _MacroRepSep_<sup>?</sup> _MacroRepOp_ )<sup>?</sup>

(I also added missing backticks around the literal parentheses tokens that delimit the group body.)

[chrisbouchard]

Suggested change

	> `$` ( _MacroMatch<sup>+</sup>_ ) _MacroRepSep_<sup>?</sup> _MacroRepOp_
	> `$` `(` _MacroMatch<sup>+</sup>_ `)` _MacroRepSep_<sup>?</sup> _MacroRepOp_

[traviscross]

We talked about this in the lang design meeting today. We were enthusiastic about this and in general agreement on the major design points.

Given that, @tgross35 is planning to pursue landing an implementation as a lang experiment. The implementation experience should help us to answer the remaining unresolved questions on some details such that this RFC can be updated to represent the likely-final user-facing behavior. With that, we can accept this RFC, and then hopefully stabilize it soon thereafter.

Thanks to @tgross35 for joining us in the meeting and for putting forward this well motivated and well written RFC.

[juntyr]

When reading, I felt the args repetition was missing a separator, e.g. ,?

Suggested change

	$pub(pub)? fn $oldname:ident( $args( $mut(mut)? $arg:ident: $ty:ty )* );
	$pub(pub)? fn $oldname:ident( $args( $mut(mut)? $arg:ident: $ty:ty ),* );

[tgross35]

@programmerjake suggested allowing the group name to $foo(a) $foo(b) to match a pattern only if the two patterns have the same number of tokens repetitions, for doing things like matching nested < ... > pairs. Discussion starting at #wg-macros > What would you say makes a Decl. Macro hard to maintain? @ 💬

[programmerjake]

I meant same number of repetitions, not tokens. so you can have $foo(a a) $foo(b) match a a a a b b

[kennytm]

allowing the group name to $foo(a) $foo(b) to match a pattern only if the two patterns have the same number of tokens repetitions

since this feature can be used to match the context-sensitive $\{ a^n b^n c^n | n \ge 1 \}$ language via $foo(a) $foo(b) $foo(c) can the macro parser actually capable of supporting this?

[programmerjake]

allowing the group name to $foo(a) $foo(b) to match a pattern only if the two patterns have the same number of tokens repetitions

since this feature can be used to match the context-sensitive { a n b n c n | n ≥ 1 } language via $foo(a) $foo(b) $foo(c) can the macro parser actually capable of supporting this?

not currently but afaict it should be relatively easy to adapt it to, assuming it was already adapted to record the group name to repetition count mapping, since for later $foo(...)* the parser just needs to make the correct number of repetitions instead of trying to parse any number of repetitions.

[corsix]

I don't necessarily want to derail this RFC, but I was contemplating the general area, and it struck me that there's a slightly different path which ends up at a similar place:

---

Extend MacroFragSpec with one new possibility, so that it becomes:

MacroFragSpec → block | expr | expr_2021 | ... | ty | vis | ( MacroMatch* )

With this, $x:(...) in a matcher declares a new metavariable $x, which captures whatever tokens the ... matches. They are captured as if they were (a sequence of) tt fragments, i.e. if the capture consists of at most one token, then using $x in the transcriber emits the matched tokens as-is rather than wrapping them in a TokenTree::Group with Delimiter::None, and if two or more tokens are matched then they are emitted just like a repeating tt fragment would be.

That's it. That's the change.

Note that:

• $x:(...) is not declaring a repetition group, so any further metavariables declared within ... are at the same repetition depth as $x.
• Kleene operators are not recognised after $x:(...). If you want a Kleene operator, then either wrap it in a repetition group (e.g. $($x:(...)),*) or put a repetition group immediately inside it (e.g. $x:($(...),*)). This is consistent with all other fragment kinds (c.f. the correct way to specify a comma-separated list of exprs is $($x:expr),* rather than $x:expr,*).

---

One building block this allows is $x:(), which declares a metavariable capturing zero tokens. You might think that such metavariables serve no purpose, but in fact they are useful for naming their enclosing repetition. I've seen people suggest this elsewhere as something like $x:empty or $x:nothing. Naming a repetition turns out to be so useful (hello this RFC) that it is worth introducing syntactic sugar for it: $x( in a matcher can be syntactic sugar for $($x:(). Just like that, we've got named capture groups, so you can write $group1( $a:ident ),+. As it's just sugar, there are no new semantics that need explaining.

Unlike the current RFC, repetition groups aren't changed to make the Kleene operator optional, nor is "full group emission" introduced, as both of these things are possible by just composing $x:(...) with existing repetition groups.

The same sugar feels appealing for transcribers: $x( in a transcriber, when $x refers to a metavariable declared as $x:(), can be sugar for $($x. Note the careful wording here: $x( already has meaning in transcribers (emit the metavariable $x, then ( is a plain token), hence the sugar only kicking in when $x refers to a metavariable declared as $x:() - as there are no such variables today, there is no behavioural change to existing macros.

Other than this sugar, if $x is a metavariable declared as $x:(), then there's nothing special about $x in transcribers. As with all other metavariables in transcribers, it simultaneously binds repetition groups in the transcriber to repetition groups in the matcher, and emits the matched tokens. It's just that the matched tokens are definitionally nothing, and emitting nothing is a no-op, hence it ends up purely doing group binding. That said, in the interest of catching mis-use, it is beneficial to make $x:() slightly special: if $x is a metavariable declared in the matcher at depth d, then it is currently an error to refer to $x in the transcriber at depths less than d (rustc will report error: variable x is still repeating at this depth), whereas for metavariables declared as $x:() it could be an error to refer to $x at depths other than d (with e.g. error: variable x is not visible at this depth). This error would not limit expressivity: if the error is reported, then $x can be moved upwards in the transcriber until it appears at the correct depth, noting that such movement is semantics-preserving (exactly because emitting nothing is a no-op).

If there is interest in catching even more mis-use, then $x( in transcribers can be slightly more than just sugar: if $x( is interpreted as sugar, then it can be an error if the resultant transcriber repetition group binds to more than one distinct matcher repetition group. As an example of this, the following is valid today:

macro_rules! zip {
    ($($x:expr),* ; $($y:expr),*) => {[$($x + $y),*]}
}

The repetition group in the transcriber binds to both of the repetition groups in the matcher. There is no great need to in this example, but let's pretend that the matcher repetition groups were given names:

macro_rules! zip {
    ($xs($x:expr),* ; $ys($y:expr),*) => {[$($x + $y),*]}
}

It would be an error to use $xs($x + $y),* or $ys($x + $y),* in the transcriber, whereas all of the following are fine and equivalent: $($x + $y),*, $($xs $x + $y),*, $($ys $x + $y),*, $($xs $ys $x + $y),*, and the following are also fine (though not equivalent): $($x),*, $xs($x),*, $xs(),*, $($y),*, $ys($y),*, $ys(),*. Again, this error would not limit expressivity, as the de-sugared form avoids the restriction.

Alternatively, if the sugar is disliked, don't do any of the sugar stuff.

---

The proposed impact on metavariable expressions feels obviously correct: ${ignore(...)} is no longer required, nor is ${count($x, depth)}. Meanwhile, ${count($x)} (without a depth) is kept as-is, and ${index(...)} / ${len(...)} can be reworked to take a metavariable rather than a optional depth number (possibly with the restriction that the $x in index($x) or len($x) must refer to an $x declared as $x:() to catch mis-use).

[programmerjake]

@corsix

With this, $x:(...) in a matcher declares a new metavariable $x, which captures whatever tokens the ... matches.

IMO the major advantage of $name(...)* as in this RFC actually naming the repetition instead of just naming the tokens inside like you proposed is that that finally removes the repetition ambiguity so you can do things like the following without getting errors like "attempted to repeat an expression containing no syntax variables matched as repeating at this depth" or "meta-variable a_v repeats 3 times, but b_v repeats 2 times":

macro_rules! cartesian_product {
    ([$a($a_v:expr),*], [$b($b_v:expr),*]) => {
        [$a($b(($a_v, $b_v),)*)*]
    };
}
assert_eq!(
    cartesian_product!([1, 2, 3], [4, 5]),
    [
        (1, 4),
        (1, 5),
        (2, 4),
        (2, 5),
        (3, 4),
        (3, 5),
    ],
);

[corsix]

Ah, I see. If that's the intended goal, then you're well beyond $x( in transcribers being describable in terms of sugar. Rather than it being an error for such a transcriber repetition to bind to more than one matcher repetition, such a transcriber repetition binds to exactly one matcher repetition (the $x one), and the binding is done regardless of depth.