[* ACTION REQUIRED *] Choosing a Core Syntax

[aphillips]

Per our discussion in the 2023-10-16 teleconference, we have narrowed the candidates for the syntax to three. These are described in this document

Following @aphillips's comment below as a general template, please stack rank your choice for the syntax to use in modifying the ABNF. Please respond before the group teleconference on Monday, 23 October 2023. Responses after that time will be ignored.

Any syntax that we choose is still subject to specific modifications using the normal group consensus process.

Of particular note, option 2a might be changed to have a non-enclosing "starter" sigil or starter character sequence instead of the enclosing sequence shown in the document. Similarly, option 3a uses a sigil %/%[ which is subject to change.

Important

"Voting" or stack ranking will be used to inform a rough consensus discussion. This is not a "winner take all" type of exercise.
We are most interested in making a good technical choice. Spend the time to elucidate your reasoning.

[aphillips]

(chair hat off)

My stack rank: 3a > 1a > 2a

• I would accept any of these syntaxes.
• I would prefer 2a to use a starter sigil instead of enclosing markers because the closing marker never has any meaning (nothing can come after the last pattern). The use of {#/#} helps alleviate the problem with up to four closing } in a row, though.
• I would prefer auto-trimming whitespace (using current consensus)

---

I prefer 3a because:

• I prefer to avoid enclosing syntax that requires users to remember and keep track of open/close markers over extended spans of message text. In 3a, all of the open/close syntax is directly around local constructs and there is no nesting.
• Unlike other syntaxes, {/} is used for exactly one thing (expressions) with exactly one syntax for expressions.
  • 1a uses {/} to delimit statements (such as input and match) and interior to these does not always preserve the expression syntax (in local)
  • 2a uses consistent expression syntax, but also uses { for multiple things (code mode, pattern quote)
• The omission of the when keyword feels like a win for authoring.

(chair hat on)

Please generally follow the above format in your own responses, although you are welcome to write as long or short as needed in the "I prefer"/"I hate" section or to common on other's responses.

[Crell]

Commentary in no special order:

• Re Whitespace, I don't feel I have a good enough grasp of the nuance to express an opinion.
• I don't hate any of them; I think all three are viable ways forward, and none would doom the spec.
• An advantage of 2a is that multi-line "codes" are naturally grouped. With 1a and 3a, every line of a code directive requires separate escaping into code mode.
• I don't feel 3a's %[foo] is a meaningful improvement over when[foo]. Given a non-coder audience, spelling out the word seems more user-friendly. (People who actually work as translators, please correct me if you disagree.)
• I think the main distinction between the options is
  • Are code blocks single line (1a, 3a) or multi-line (2a)?
  • Are code blocks beginner-delimited (3a) or wrapped (1a, 2a)?
• (Where multi-line implies wrapping)

So I think my ranking would be:

3a == 2a > 1a

I'm torn on multi-line code blocks vs single-line code blocks. However, I see the advantage of single-line blocks being only beginner-delimited, so if we go single-line code, that seems reasonable.

Also, regarding the sigil to use... Another option could be //. Yes it means "comment" in almost every programming language, but that's also true of #, and // at least is extremely rare in human-text so the odds of it needing to be escaped are very small. (This would only really work as a begin-only delimiter in option 3a.)

[eemeli]

1a > 3a > 2a

I really like the first-order simplicity of 1a, where all the text is outside curly braces, and all the special stuff is inside them. I honestly think that this is the best or least-worst option when considering translators who end up working directly with messages in MF2 syntax, rather than through any purpose-built tooling. The only place where 1a nests braces is within the match, and that we could get rid of by requiring multiple separate statements for the rather rare case of multiple selectors.

I think 2a is two syntaxes in a trenchcoat. I think it'll lull many developers to getting familiar-ish with the first single-pattern syntax while ignoring the second, and make it that much more likely to not pluralize messages that really ought to be pluralized, i.e. use Message count: {$n} rather than {$n} messages because the former doesn't need code mode. Because single-pattern messages are way more common, it'll mean that on variant messages a reader of the syntax needs to realise that:

1. with {# or {{ we're entering this special "code mode" that changes everything,
2. after the curly braces of the match {…}, curly braces aren't wrapping code, but patterns,
3. then we're back to the single-pattern syntax, except that here it ends in another closing curly brace,
4. and then after the final #} or }} the message must end, rather than continuing like it does in e.g. MF1 and Fluent.

A little bit of 2a's excessive complexity could be trimmed off by dropping the terminal #}, but that won't get rid of the conceptual syntax levels that @stasm illustrated rather well. It also relies quite heavily on the code-first approach we've been looking at for a while, so I rather doubt that it'll let us get rid of the doubled or tripled meanings that get assigned to curly braces, given that we'd not managed to do so previously.

I think 3a is better than 1a in differentiating statements from placeholders by using the % prefix for them. This does introduce a few drawbacks, however:

1. It's harder to spot that statements are mostly delimited by %…}, and that doesn't work for multi-selector variants. So to keep track of "code" vs "text" you need to remember the syntax for each statement, unless they're each on a separate line.
2. Because % is so common, we need something rare like the %[ sequence to start the "when", and that means we need some sigil to communicate what the other syntaxes use the word when for, and that's a bit unfortunate.

Otherwise, conceptually, 1a and 3a are pretty close to each other.

---

When comparing the formats, the message I've mostly been staring at is the simple-selector message, because I figure that'll be the most common not-simple message, and therefore it matters the most. And for that, the minified single-line syntax provides the most differentiation:

{#match{$foo :plural}}{#when foo}Hello {$foo} you have a {$var}{#when *}{$foo} hello you have a {$var}
{#match{$foo :plural}when foo{Hello {$foo} you have a {$var}}when *{{$foo} hello you have a {$var}}#}
%match{$foo :plural}%[foo]Hello {$foo} you have a {$var}%[*]{$foo} hello you have a {$var}

My greatest stumbling block when reading the above is catching that for 2a the first when foo is code and not text. Next up, it's separating out {$var}%[*]{$foo} into its constituent parts, and noticing in particular that the % is code and not text, attaching to the [*] and not the {$var}.

I don't really mind the character length of 1a being greater than the others. What's most important to me is the amount of stuff I need to mentally remember and track, and that to me is minimal in 1a.

One "ha ha, only serious" possibility for 3a would be to use the already escaped \ as the statement sigil, so we'd have e.g. \input and \when. This would not add any new restrictions on patterns, and would more clearly than any other choice conceptually attach to whatever follows. I mean, as it works for LaTeX, it could work for us as well? 😇

\match{$foo :plural}\when[foo]Hello {$foo} you have a {$var}\when[*]{$foo} hello you have a {$var}

---

My understanding is that we're now picking a general syntax direction so that we have a baseline from which to consider further questions, such as:

• How to treat external whitespace, and whether we need to revisit the previously established consensus on this.
• Whether statements other than when should be combined into a single preamble code block.

In addition to the negatives I've listed above, I dislike 2a because it forces us to decide about all these things at the same time, rather than allowing us to make stepwise progress. Both of the above concerns may be addressed later within a syntax based on either 1a or 3a. If we try to pick 2a, we'll need to resolve not just the general syntax direction but also at least the above issues before being able to make that decision.

[vdelau]

1a > 3a > 2a

For me 1a makes a lot of sense, although I'd revisit and simplify the whitespace quoting:

• Either quote the whole pattern, preserving all whitespace inside, or have a 'bare' pattern where leading and trailing whitespace is trimmed. I find the 'quoted' literal syntax ({| / |}) to be very noisy and hard to parse.
• When trimming, treat all whitespace equal, not only ASCII whitespace.
• Personally, I'd also preserve whitespace in simple patterns that do not contain code besides placeholders, but for consistency I can imagine that whitespace would be trimmed. The reason for not doing this is that for automatically collected pattern, which would in that case be simple, will preserve whitespace.

I do prefer the concept of 2a over 3a, explicitly entering code mode. However, I dislike the code mode syntax as per Eemeli's reasoning above. I also have a hard time parsing that as a human, especially the when syntax. I think it was better with the sigil prefixes in one of the previous proposals, in combination with enclosing the when parameters in some form of brackets. I would prefer a marker only at the start of the message, which should only be optionally preceded by whitespace.

I do like option 3a in the sense that using a sigil instead of braces would work for me and could result in a cleaner look. However, I'd suggest to not abbreviate away the when. The same whitespace quoting concerns as mentioned at 1a apply.

[stasm]

My understanding is that we're now picking a general syntax direction so that we have a baseline from which to consider further questions, such as:

• How to treat external whitespace, and whether we need to revisit the previously established consensus on this.
• Whether statements other than when should be combined into a single preamble code block.

I agree that these are the two currently most impactful axis of decisions. (With the first one being a proxy for the decision between the triple-layer and dual-layer models.) And I think they are, in fact, the direction that we're looking for. It seems, however, that you'd like to consider them afterwards, which leaves me puzzled. What is the direction that we're picking now in this case?

I only thought about it today, but I wonder if it would be more helpful to vote on a matrix representing the above two questions. 1a and 3a cover one cell (trim whitespace, separate statements), 2a covers another (don't trim whitespace, group statements in a preamble), and we'd need to define what the other cells entail.

In addition to the negatives I've listed above, I dislike 2a because it forces us to decide about all these things at the same time, rather than allowing us to make stepwise progress. Both of the above concerns may be addressed later within a syntax based on either 1a or 3a. If we try to pick 2a, we'll need to resolve not just the general syntax direction but also at least the above issues before being able to make that decision.

Arguably, the same point can be made about 2a. It can also evolve to incorporate ideas from 1a/3a, most notably, the dual-layer model.

(A longer comments coming soon.)

[aphillips]

@eemeli

I find it amusing that you're unhappy with 3a because it removes when--which was your idea. If we go back to 3a with when we get:

%match{$foo}%when{foo}Hello {$foo} you have a {$var}%when{*}{$foo} hello you have a {$var}

@stasm noted:

I wonder if it would be more helpful to vote on a matrix representing the above two questions.

I don't think it would. We need a syntax. The various whitespace and organizational hiccups are inherently entangled with the syntax. We can pretend to discuss them separately, but the choices we make depend, fundamentally, on details of the message grammar.

I'll point out again what I've said elsewhere: the whitespace problem can be done away with by requiring the code-internal pattern to be quoted. So:

{#match{$foo :plural}}{#when foo}{Hello {$foo} you have a {$var}}{#when *}{{$foo} hello you have a {$var}}
{#match{$foo :plural}when foo{Hello {$foo} you have a {$var}}when *{{$foo} hello you have a {$var}}#}
%match{$foo}%when{foo}{Hello {$foo} you have a {$var}}%when{*}{{$foo} hello you have a {$var}}

Some syntaxes can have code preambles (blocks) and others won't make sense with them. But each is a variation of a given syntax. Let's pick one so that we can go back to what works in this WG: concrete changes to the ABNF.

[stasm]

I struggle to cast a definitive vote because there are many latent decisions in each of the proposals. Voting is polarizing, and I tend to think that the final solution should instead try to combine multiple good ideas from each of the proposals. Furthermore, I'm optimistic in that I think that the final syntax can be derived from any of the 3 currently discussed ones.

My priorities are:

• Build on a sound mental model, which can be explained to and deduced by users.
• Optimize for the single line representation.
• Prefer keywords and avoid the proliferation of sigils, as they are reduce the discoverability of the syntax and are difficult to even search for.
• Allow unquoted variant patterns.
• Discourage the idea of putting text around the match statement, or putting more than one match in a single message (both are errors).

---

If we're voting on the general approach, or the mental model for what happens to variant patterns (see the illustration in #496 (comment)), then 1a/3a > 2a.

• The triple-layer model should be optional (via {{...}}), and the dual-layer model should be the default.
• Simple unquoted messages should not be trimmed, while unquoted variant pattens should. This is inconsistent on purpose -- in my talking to people outside the WG this seemed to be the least surprising behavior.
• As they currently stand, I think both 1a and 3a can be improved when it comes to their specific choices for sigils and otherwise.

If we're voting on specific look & feel of the syntax, then 2a > 3a > 1a.

• Keywords should not be prefixed with new sigils.
• Statements should be grouped together.
• As an extra benefit, 2a requires the least amount of changes to the current ABNF, which makes it a good choice for the baseline of further syntax considerations.

---

If I could extend the comparison table with subjective reasons, I'd list the following pros and cons:

In 1a:

• (+) It looks like a templating language.
• (+) No new escape sequences.
• (-) {...} is used for multiple concepts.
• (-) Keywords are introduced with a new sigil, making the input statement (hopefully relatively common in messages with placeholders) be spelled as #input $var :func -- that's 3 words and 3 different sigils, which I'd like to avoid as much as possible.

In 2a:

• (+) Keywords are not prefixed with extra sigils.
• (+) Statements (input, local, match) form a coherent block at the beginning of the message body.
• (+) The match ... when sequence is not broken up by delimiters.
• (-) It's two syntaxes within one.
• (-) The fact that the code-mode marker requires the closing part may suggest that {# #} is a kind of a placeholder which can be have text around it.

In 3a:

• (+) {...} is only used for expressions.
• (+) Keys stand out better between patterns thanks to the use of square brackets.
• (+) Variant patterns can, but don't need to be quoted.
• (-) Introduces a new sigil for keywords.
• (-) Statements are not delimited.

---

I think the final solution should combine most of the pluses from the above list. In essence, this boils down to combining the code-mode block from 2a (in form of a preamble) with the dual-layer model for variants from 3a. For instance:

{# input {$date :datetime dateStyle=long} #} Today is {$date}.

{# input {$count :number} match {$count :plural} #} {[1]} One thing. {[*]} {$count} things.

[Crell]

@stasm I'm unclear in your final example, where does the match block end? Can anything come after "things.", and how do we know which is which? It looks like you're out of code mode there, but still within the conceptual match block, which is confusing to me.

[samdark]

Either 1a or 3a are good. I think there will be an issue with foced multiline translation strings as @eemeli pointed out.

[eemeli]

@stasm:
I agree that these are the two currently most impactful axis of decisions. (With the first one being a proxy for the decision between the triple-layer and dual-layer models.) And I think they are, in fact, the direction that we're looking for. It seems, however, that you'd like to consider them afterwards, which leaves me puzzled. What is the direction that we're picking now in this case?

I think we're picking a general shape for the syntax, with a specific form to start, from which we may iterate further. I would classify the choices as:

1. Everything non-text is in curly braces.
2. Separate "simple" and "complex" syntaxes, with some toggle or special wrapper for the latter.
3. Curly braces are always expressions; statements use a different code-y syntax.

@aphillips:
I find it amusing that you're unhappy with 3a because it removes when--which was your idea. If we go back to 3a with when we get:

%match{$foo}%when{foo}Hello {$foo} you have a {$var}%when{*}{$foo} hello you have a {$var}

That is a better syntax in many ways, but it does require escaping errant % signs in patterns due to the %when being special. As I put it earlier, it is unfortunate that we can't really do that. I'm pretty sure that \ is the only single-character sigil we could use for that, and that's why I did mention it in my previous comment, with the \when.

[Crell]

The problem with \ is that it's also the escape character in many languages' own string syntax, so could easily get interpreted as \w followed by hen. Which is likely not the intent.