Discussion thread for code mode introducer

[aphillips]

This is where you can freely discuss technical issues related to the ballot in #525

[eemeli]

Out of interest, I went and looked at the frequencies with which the sigils suggested for Option D .~@&%# show up as first characters in the Mozilla and Android message corpuses, when followed by some a-z characters. Some observations:

• Full Stop U+002E .
  In Android, there are two messages in one file that do not include an initial . in English, but do in some translations start as .vcf. One other Android message also starts with .vcf in English.

• Tilde U+007E ~
  No messages found starting with ~[a-z].

• Commercial At U+0040 @
  Numerous hits for both Android and Mozilla. In the former, it's a special reference operator. In the latter, mostly in messages referring to CSS.

• Ampersand U+0026 &
  Numerous hits for both Android and Mozilla, mostly as it's an HTML/XML entity introducer.

• Percent Sign U+0025 %
  On both, numerous hits for the printf variables %s and %d in particular. When looking for % followed by at least two a-z characters, a few hits in both conjugating %s variables with suffices, mostly in Uzbek. Many hits in Mozilla due to some systems supporting %varname% placeholders. All instances are due to placeholder syntax, which is different in MF2.

• Number Sign U+0023 #
  Many hits on Android with RGB colors like #ffff00, and some on Mozilla for hashtags like #espacios-mozfest-2022.

So if going this way, I think . and ~ are the reasonable candidates to consider. @ shows up a bit too often, & is used for HTML entities, and % has rather established usage as a variable placeholder introducer. # is surprisingly common, and is used as a line comment introducer in too many host formats.

One negative for ~ is that on some keyboards (at least Italian) it's not necessarily directly available at all.

If we go with ., we may want to require escaping it only if it's followed by at least four a-z characters. This would avoid the likely most common conflict with .abc three-letter file extensions, while allowing later potential expansion of the supported commands beyond input/local/match.

[aphillips]

In #521 @macchiati made this comment, which I copy here for visibility and because I am closing that thread. Note that @duerst also made a comment about non-ASCII there, which I'm not moving over.

---

It seems to me that there is a higher-level breakdown, where 'special sequence' below is a sequence of one or more ASCII punctuation/symbol characters.

Alpha: mark the start of a complex message with a special sequence, and either the end of the declarations or the the end of the complex message with a special sequence (Option A, F)

Beta: mark the start of a complex message with a special sequence (Option B, C, E)

Gamma: mark start of each keyword that starts statements with a special sequence (Option D)

Personally, I don't think either Alpha or Gamma add much value; and the key to Beta is to pick a sequence that is distinctive, doesn't collide with any other sequence, and very unlikely to start a simple message (thus probably a multi character sequence of symbols instead of a single symbol.

• By the way, we'd have a lot more choices for these if we considered non-ASCII — §§ would be a nice separator.

---

[aphillips]

@macchiati I think that's one way one could break down the options.

I think your "Alpha" would be more succinctly put as: "enclose all or part of a complex message with a special sequence". There's nothing wrong with this (many many syntaxes feature enclosure--our own syntax encloses both patterns and expressions [placeholders] already). But my feeling here is that the enclosure is one layer too many.

My preference for "Gamma" (option D) is precisely that it adds nothing to the syntax. One has to learn the keywords anyway: ours are just spelled funny. It's easy to understand because there's only one concept.

I could be okay with "Beta", but the challenge (as Mark points out) is finding the right sequence. My concern would be that I don't see what the value is to the user with B/C/E (except, of course, that the message parses correctly). It's a piece of extra gunk that we require everyone to type in order to serve the parser.

@stasm Proposed option F, which is a preamble syntax, and he's expressed in teleconference (and elsewhere) that it's still underdeveloped as a proposal. I like the idea of having a conceptual model for messages that make it easier to manage the code and patterns. I also like the idea of getting rid of when.

On the other hand, I don't see how the preamble concept works by dividing match from the variant map. The unknown arity of match was cited as a reason to delimit the match statement, but with a LL(1) or LL(2) parser, we're not seeking the close marker and the key syntax does just as good a job of closing the arity. The expressions in a match are already chock full of sigils, so adding noise to that line feels painful:

{% match {$numUsers :plural} {$numDogs :plural} {$activity :select} %}  <- is this last token adding anything?
0 * * {{Nobody did anything with their dogs}}
* * * {{Everyone did everything with their dogs}}

I think it's useful to spend time writing messages in a syntax to get a feel for it. Thinking about message in the abstract doesn't tell me much finger knowledge or "feel" of the syntax compared to trying to do different things.

[eemeli]

On the other hand, I don't see how the preamble concept works by dividing match from the variant map. The unknown arity of match was cited as a reason to delimit the match statement, but with a LL(1) or LL(2) parser, we're not seeking the close marker and the key syntax does just as good a job of closing the arity. The expressions in a match are already chock full of sigils, so adding noise to that line feels painful:

{% match {$numUsers :plural} {$numDogs :plural} {$activity :select} %}  <- is this last token adding anything?
0 * * {{Nobody did anything with their dogs}}
* * * {{Everyone did everything with their dogs}}

My preference would be to use multiple match statements where they're necessary, with one selector per statement. That way, we could also drop the braces from the selector. Using a D-ish syntax, the message above could look like this:

.match $numUsers :plural
.match $numDogs :plural
.match $activity :select
0 * * {{Nobody did anything with their dogs}}
* * * {{Everyone did everything with their dogs}}

A vast majority of messages with selectors will only ever have one selector; for those, this change would let us get rid of one {} pair. For messages with multiple selectors, this would make it clearer what is being matched.

[stasm]

Option F proposes to shift from thinking about match as a statement which continues onto the variants. Our current syntax is deeply rooted in this concept: it's a switch-case in disguise. I don't think it's bad at all, but I like that F introduces something new conceptually, and it's appealing to me.

I see F as a solution to the problem of having all this additional stuff that we want to decorate the message with. Selectors, input, locals... where do we put it all? We only have the message itself to work with, so it needs to be in-band.

Instead of solving this through nesting (text → code → text), F solves it with the preamble (text → code; text). The preamble is where we declare inputs, locals, and selectors.

It's a bit as if the message was a map with a semantic comment:

/*
 * @input $count :number
 * @match $count :plural
 */
{
   one: "One thing"
   default: "Many things"
}

---

On the other hand, I don't see how the preamble concept works by dividing match from the variant map. The unknown arity of match was cited as a reason to delimit the match statement, but with a LL(1) or LL(2) parser, we're not seeking the close marker and the key syntax does just as good a job of closing the arity.

You're right, parsers will do fine without an explicit closing delimiter for the preamble; that's how it works right now after all. I meant that the closing delimiter can help people understand how the message is structured.

On a related note, I'm not fond of the fact that both input and local have the arity of 1, but match has >1. I filed #527.

[stasm]

My first preference is for F, but I also see benefits to A which I'd like to make sure we don't omit.

The closing }} of A doesn't add value as long as we're only considering a single message. However, looking at the larger picture, the closing delimiter would be very useful for designing the resource format.

key-simple = A simple message
key-complex = {{
    input {$count :number}
    match {$count :plural}
    when one {{One thing}}
    when * {{Many things}}
}}
another-key = {{ {{Pattern}} }}

We'd need to figure out the exact syntax and whitespace handling for simple messages (perhaps wrap them in {{ ... }}, same as variant patterns?). This, in turn, would imply that we can't use {{ ... }} for message delimiters. We could look for a different double sigil, like {% ... %}, but I think that we could also win a lot by combining A with D:

key-simple = A simple message
key-complex = {
    .input {$count :number}
    .match {$count :plural}
    one {{One thing}}
    * {{Many things}}
}
another-key = { {{Pattern}} }

This way we could use a single brace, followed by a .keyword or a quoted pattern, as the code-mode introducer. Single curlies are universally recognized as block delimiters. Furthermore, the simple/complex duality is also a concept familiar to many developers. For example, JavaScript functions can be spelled as () => value or as function() { return value; }.

[eemeli]

The closing }} of A doesn't add value as long as we're only considering a single message. However, looking at the larger picture, the closing delimiter would be very useful for designing the resource format.

Actually, it's the other way around. Consider for instance this YAML representation of your example:

key-simple: A simple message
key-complex: '{{
    input {$count :number}
    match {$count :plural}
    when one {{One thing}}
    when * {{Many things}}
}}'
another-key: '{{ {{Pattern}} }}'

First of all, many such formats consider {} as mapping keys, in particular when it's used as the first character of a string: This is why the second and third messages need quotes.

Second, that's actually invalid YAML. Can you see the error?

[stasm]

First of all, many such formats consider {} as mapping keys, in particular when it's used as the first character of a string: This is why the second and third messages need quotes.

Can you give a few other examples, besides YAML?

Second, that's actually invalid YAML. Can you see the error?

I don't know YAML well enough to be good at parsing it visually.

[aphillips]

@stasm mentioned:

You're right, parsers will do fine without an explicit closing delimiter for the preamble; that's how it works right now after all. I meant that the closing delimiter can help people understand how the message is structured.

Totally. For me, people's "association" between match and the flock of variants is what is kind of important.

@eemeli suggested:

.match $numUsers :plural
.match $numDogs :plural
.match $activity :select
0 * * {{Nobody did anything with their dogs}}
* * * {{Everyone did everything with their dogs}}

What I don't like about this is that the match statements are laid out vertically while the keys are horizontal.
It is also tempting to reorder the match statements, which would break the selector (significantly change its meaning)

I agree with @stasm that match is just a kind of switch/case statement (perhaps in a tuxedo??). I don't think that's a bad thing, in that the keys can be visually aligned with the selectors or are at least easily paired up

[eemeli]

First of all, many such formats consider {} as mapping keys, in particular when it's used as the first character of a string: This is why the second and third messages need quotes.

Can you give a few other examples, besides YAML?

At least some JSONC variants and Hjson will allow for some messages to go unquoted, but will require quotes if they start with {.

Second, that's actually invalid YAML. Can you see the error?

I don't know YAML well enough to be good at parsing it visually.

The error is that the last line of the key-complex value is not sufficiently indented. In a block-map, all lines of a value need to be at least one space more indented than their parent key. If our syntax ends up with {...} around it, this will make embedding MF2 messages in YAML a pretty bad experience because people have a natural tendency to align the closing brace with the same level of indentation.

This is effectively one of the corner-cases of YAML, and it will not be caught by all YAML processors. The way that e.g. the syntax highlighter here doesn't catch the error will make the experience even worse.