Clarify that standalone markup is permitted.

[aphillips]

Is your feature request related to a problem? Please describe.
From our Slack discussion of 2023-02-18/19/20, our documentation should be clear that we allowed unpaired markup.

Describe the solution you'd like
See above

Describe why your solution should shape the standard
It's fundamental to parsing/validation

Additional context or examples
Pasting the Slack conversation:

Agree about empty string.
We currently do not allow unpaired markup (we say so explicitly). This is not consistent with HTML, as you note, but I don't know what went into that decision.
+1000 on fuzzing.

Mihai
2 days ago
I argued many times that markup dies not have to be paired. And that we don't need markup at all :-)

Mihai
2 days ago
s / dies not / did not /

Mihai
2 days ago
This is the problem with landing things where we have no agreement.

Mihai
2 days ago
What went into that decision was time pressure, and the spec landed 2 days before people were living for the summer. It was either that, or miss the ICU release. (edited)

Mihai
2 days ago
I didn't implement markup in the ICU tech preview, and I have an open issue.
But this is what I tried to warn about in the last meeting. Agree that in a somewhat unarticulated and emotional.
But I was seeing it happening again right there

Mihai
2 days ago
Maybe in spec we should mark things that are not agreed with a special tag.
Otherwise "we decided", and changing them requires more effort. You are "challenging an existing decision" (and in that meeting there was also a push to make that harder)

Mihai
2 days ago
And "just file an issue" is not a solution. Give it some time and it becomes "Mihai is challenging a decision" instead of "we submitted with disagreement".
All that we have as a recording is people remembering.

Mihai
2 days ago
(meaning: not much)

Mihai
2 days ago
Sorry, I should not bring these "heavy topics" over a (long) weekend, over chat, and at unfriendly hours for people in other time zones.
I will stop, and choose another avenue
👍
1

Eemeli
21 hours ago
We currently do not allow unpaired markup (we say so explicitly). This is not consistent with HTML, as you note, but I don't know what went into that decision.
Where do we say so explicitly? Doesn't this section say the opposite, as it states that "[markup elements] do not require well-formedness"?

Staś Małolepszy
10 hours ago
Yeah, my impression was also that unpaired markup is fine. My comment above about the unpaired {-m} being invalid was wrong, sorry about that
👍
1

Addison Phillips
4 minutes ago
Non-well-formedness means lots of things. For example, it can mean cross-over tags:
some {+a}{+b}mixed tags{-a}{-b}

Addison Phillips
3 minutes ago
We don't show standalone tags anywhere in the examples nor do we explain that they are permitted. I'm pretty sure I saw paired-only somewhere recently in our docs, hence my comment.
If we mean to allow standalone tags, we should say so explicitly.

[mihnita]

I think that we should not have markup at all.
It is not needed, and the current spec for markup is hard to use to support more than one kind of markup. And that is needed.
We had long and heated arguments about it.

All the formatting can be handled with placeholders.

Markup open issues:

• Markup: do we need a different sigil for display element names? #241
• Do markup elements map to registered functions? #262
• Markup: how do ensure non-HTML display elements are first class? #238

And to show that it was not agreed on, see PR #283 in Jun 21, 2022:

I will approve it to move things forward and not have these many open PRs.
It does not mean I agree with this syntax, so it should not be hold against me 1 month from now ("but you approved X")
There are still at least 2 open issues arguing about how to best handle "markup elements"

[mihnita]

We currently do not allow unpaired markup (we say so explicitly).

This goes directly against the Unicode TC decision: "Should not assume well-formedness of elements"

[echeran]

This issue is interesting because it touches on a mix of technical and process/meta-level concerns. Although many are intertwined, so maybe we can discuss how to discuss?

Technical: I agree that we should call out standalone tags as explicitly allowed with examples. Although that brings up the question of syntax for them, and the higher-level question of what benefits we gain for entertaining special syntax for markup.

Meta: We had an issue to discussion how to represent standalone markup tags (#239), but it got closed in favor of #238, which is still open. We also had issue #269 about sigils for markup tags, which was closed after PR #283 was merged, even though it handles open/close tags only. That leaves us in this curious state where open/close markup tags have special syntax in the current spec, and I think standalone markup tags use regular placeholder syntax.

Meta/Technical: We have multiple open issues about various details of markup syntax, but we still haven't addressed the question of what special syntax for markup tags buys us in the first place, on top of just using regular placeholder syntax. I asked about this in #262, which is still open. I don't recall hearing a thorough & compelling answer so far.

We had long and heated arguments about it.

@mihnita I don't think the length or emotional content of arguments imply the importance of the conclusion, but rather, the technical merits ought to be the main determinant, and I think we have been doing well in returning to the technical points and avoid non-technical feelings and distractions. It might be good for us to consider things in terms of tradeoffs and asking pros & cons, and maybe evaluative criteria on top of that, if arguments get repetitive.

[aphillips]

Let's discuss how to approach this Monday.

Some specifications have features that are "at risk" and I think markup would qualify as an "at-risk" feature in our spec. We have proposed a syntax to support it, but don't have implementations or clarity on requirements and functionality. This is a recipe for problems when we discover those requirements later...

Some approaches suggest themselves:

1. We could avoid defining markup support now, but reserve a number of sigils and tokens for future standardization that could include markup support. This would allow us to avoid having an at-risk feature in our specification while preserving the ability to add support for it later. Processors could optionally swallow such placeholders if they did not understand them or optionally warn/error on them. We would probably want to add more sigils than just + and -.

2. We could provide a "private use placeholder" syntax which implementations could use to implement markup or some other scheme (for example TTS management). Processors could optionally swallow such placeholders if they did not understand them or optionally warn/error on them.

3. We could adopt an approach like XLIFF, in which any markup is part of the prose, but we provide placeholders to signal to the translation system what to protect and provide metadata. XLIFF takes the approach that markup in content is just string data which can be protected in various ways, notably <ph> for standalone placeholders and <bpt>/<ept> for paired placeholders (where the pair need to appear in the translation and be in the same order, even if the contents between them changes and the position in the string changes).

I think (3) is interesting because one of the challenges with markup is that for Web platforms there are often two levels of it. There are HTML tags and then there are templating languages used by page generation frameworks. There are an unreasonable number of templating languages, many homegrown. If we took an XLIFF approach that might be something we could accomplish and build implementation around (since we don't actually have to build support for pluggable markup regimes). However (1) and (2) are also viable (and these options are not necessarily mutually exclusive--we could do combinations of these or all three).

[eemeli]

To help in our deliberations, I'd like to submit the following three real-world Fluent messages to be used as examples that I believe we should be able to support in MF2. These are all from preferences.ftl, lines 92, 125 and 843:

## Extension Control Notifications
##
## These strings are used to inform the user
## about changes made by extensions to browser settings.
##
## <img data-l10n-name="icon"/> is going to be replaced by the extension icon.
##
## Variables:
##   $name (string) - Name of the extension

extension-controlling-password-saving =
  <img data-l10n-name="icon"/> <strong>{ $name }</strong> controls this setting.

## Preferences UI Search Results

search-results-help-link =
  Need help? Visit <a data-l10n-name="url">{ -brand-short-name } Support</a>

## Firefox Account - Signed out

# This message contains two links and two icon images.
#   `<img data-l10n-name="android-icon"/>` - Android logo icon
#   `<a data-l10n-name="android-link">` - Link to Android Download
#   `<img data-l10n-name="ios-icon">` - iOS logo icon
#   `<a data-l10n-name="ios-link">` - Link to iOS Download
#
# They can be moved within the sentence as needed to adapt
# to your language, but should not be changed or translated.
sync-mobile-promo =
  Download Firefox for <img data-l10n-name="android-icon"/> <a data-l10n-name="android-link">Android</a> or
  <img data-l10n-name="ios-icon"/> <a data-l10n-name="ios-link">iOS</a> to sync with your mobile device.

In the above, the <a> and <img> with data-l10n-name attributes are DOM elements for which other attributes (such as src and href) are provided elsewhere, while <strong> is inline markup that's a part of the message.

According to my understanding of how our current markup syntax ought to work, I believe this could be a representation of the above in MF2, presuming that appropriate :brand, :img, +a, and +strong are available:

extension-controlling-password-saving =
  {{:img key=icon} {+strong}{$name}{-strong} controls this setting.}

search-results-help-link =
  {Need help? Visit {+a}{(short-name) :brand} Support{-a}}

sync-mobile-promo =
  {Download Firefox for {:img key=android-icon} {+a key=android-link}Android{-a} or
   {:img key=ios-icon} {+a key=ios-link}iOS{-a} to sync with your mobile device.}

[aphillips]

@eemeli noted:

According to my understanding of how our current markup syntax ought to work, I believe this could be a representation of the above in MF2, presuming that appropriate :brand, :img, +a, and +strong are available:

Your description of the current "theory of markup" sounds correct to me. That presumption at the end is the nub of the problem. How does one ensure that +a and +strong are available?

Presumably, the runtime that supported your HTML markup would consume the {+mumble} and {-mumble} tags and produce the correct string output. This could take the form of code that reads the input and produces (perhaps very) different markup as output.

For example, it wouldn't be unreasonable to have style in your source:

Visit <strong style="text-alignment: left; background-color: pink">here</strong>

That requires extra processing to make into valid MF2, because the quoting and perhaps other stuff has to be replaced with our syntax, such as replacing double-quotes with our literal markers:

{Visit {+strong style=(text-alignment: left; background-color: pink)}here{-strong}

Also, note that your examples do not include abusive use, e.g.:

{I am the starting string <strong>}
...
{and</strong> I expect to be concatenated to the starting string}

Finally, your examples are strictly in a single markup dialect. There are lots of pages assembled by what I called a "templating language" (think of stuff like mustache) where the string contains both HTML and "something else". The HTML markup handler wants to generate </>. Mustache, for example, uses {{ and }} for it's template commands. It's unreasonable to expect that people will not use their templating language in strings, such as:

Click <a href={{ myUrl }}>here</a> to do something {{ descriptionValue }}

Alas, what happens when the HTML and template language clash:

Click <a href="mumble">{{# strong }}here{{/ strong }}</a> to do something

=>

{Click {+a href=(mumble)}{+strong}here{-strong}{-a} to do something

How does the MF processor know which handler gets the keyword strong?

I suspect we might be better off keeping MF out of the markup management business in favor of the markup protection business (using something vaguely XLIFF-like for the example):

{Click {+bpt id=a}<a href="mumble">{-bpt}{+bpt id=strong}\{\{#strong\}\}
    {-bpt}here{+ept id=strong}\{\{/strong\}\}{-ept}{+ept id=a}</a>{-ept} to do something}

(Of course that is a lot of work for the developer and fairly unattractive...)

[zbraniecki]

We played today with the difference between Elango's and my mental model and arrived to a reduced discrepancy:

With Markup as a AST node:

Open {+html:a}Firefox{-html:a}, which is a {+html:a}Mozilla{-html:a} {+ssml:loud}product{-ssml:loud}.

With markup as a placeholder node:

Open {:markup type="open" tag="a" ns="html"}Firefox{:markup type="close" tag="a" ns="html"}, which is a {:makup type="open" tag="a" ns="html"}Mozilla{:markup type="close" tag="a" ns="html"} {:markup type="open" tag="loud" ns="ssml"}product{:markup type={"close" tag="loud" ns="ssml"}.

They're not that different!

[eemeli]

@aphillips:
Your description of the current "theory of markup" sounds correct to me. That presumption at the end is the nub of the problem. How does one ensure that +a and +strong are available?

I would presume that we would include in the registry an enumeration of the supported markup elements, and their valid options.

Presumably, the runtime that supported your HTML markup would consume the {+mumble} and {-mumble} tags and produce the correct string output. This could take the form of code that reads the input and produces (perhaps very) different markup as output.

While that's a possibility, I do not expect that in real-world use a runtime would necessarily be reproducing some other intermediary string representation of a message including markup, but rather building the appropriate output directly. Taking my first example message as a starting point:

{{:img key=icon} {+strong}{$name}{-strong} controls this setting.}

When formatted for display, this makes much more sense to be formatted as parts, rather than a string, resulting in something like:

[
  Image(key='icon'),
  Literal(' '),
  MarkupStart('strong'),
  Literal('ExtensionName'),
  MarkupEnd('strong'),
  Literal(' controls this setting.')
]

If the target is an HTML DOM as in the original, the Image would be matched with a separately defined <img>, and something like document.createElement('strong') used to handle the MarkupStart/MarkupEnd and its body.

That requires extra processing to make into valid MF2, because the quoting and perhaps other stuff has to be replaced with our syntax, such as replacing double-quotes with our literal markers:

That's correct, and unfortunate. My personal preference would have been for us to support XML syntax for markup, but this was not supported by the whole group.

Also, note that your examples do not include abusive use [and] are strictly in a single markup dialect.

That's right. I am not trying to claim that these are exhaustive, but they are actual messages we will want to be able to represent in MF2. It would be very useful if others could provide real-world examples from other sources which e.g. break start/end markup across messages; Mozilla's Fluent messages don't have any such.

Alas, what happens when the HTML and template language clash:

Click <a href="mumble">{{# strong }}here{{/ strong }}</a> to do something

=>

{Click {+a href=(mumble)}{+strong}here{-strong}{-a} to do something

How does the MF processor know which handler gets the keyword strong?

If this is handled within MF2, then I would expect the function registry to provide the answer. But that only works if the target is an intermediate string representation, which would require subsequent parsing. If the target is a more final shape, then MF2 only needs to provide a parts representation that'll be handled by some next phase that determines what to make of the markup parts.

I suspect we might be better off keeping MF out of the markup management business in favor of the markup protection business (using something vaguely XLIFF-like for the example):

{Click {+bpt id=a}<a href="mumble">{-bpt}{+bpt id=strong}\{\{#strong\}\}
    {-bpt}here{+ept id=strong}\{\{/strong\}\}{-ept}{+ept id=a}</a>{-ept} to do something}

(Of course that is a lot of work for the developer and fairly unattractive...)

Yes, that would be rather unergonomic.

[echeran]

It would be very useful if others could provide real-world examples from other sources which e.g. break start/end markup across messages; Mozilla's Fluent messages don't have any such.

We use a tooling layer between source documents and messages that generates our messages in a format-independent way. It leverages Okapi for its many benefits in this regard (ex: support for many formats & types of markup out of the box). The handler for the format/markup determines how to create text units and segments. It performs segmentation on the text units using ICU4J BreakIterator to subdivide them into segments, which correspond to individual messages.

So in the case of an HTML markup document, if it contains something like:

<span ...><i>You're viewing <b>Apigee X</b> documentation. <br/> View <a href="..." title="..." ...>Apigee Edge</a> documentation.</i></span>

then we ultimately might get 4 segments that are the equivalent of MF2.0 messages in our current syntax looking like:

{{:markup tag="span" type="open" ns="html" ...}{:markup tag="i" type="open" ns="html"}You're viewing {:markup tag="b" type="open" ns="html"}Apigee X{:markup tag="b" type="close" ns="html"} documentation.}

{{:markup tag="br" type="standalone" ns="html"} View {:markup tag="a" type="open" ns="html" href="[#$3]" title="[#$4]" ...}Apigee Edge{:markup tag="a" type="close" ns="html"} documentation.{:markup tag="i" type="close" ns="html"}{:markup tag="span" type="close" ns="html"}}

{https://the.url.from.the.a.tag/}

{The title text from the a tag}

Okapi will also assign unique ids per placeholder and unique ids per placeholder pair, which I'm eliding here because: 1) brevity 2) we hide from the translator such information in the CAT tool anyways, since it doesn't concern them

So it's natural for someone new to seeing this to question whether this is all a good thing, or if it is all too complex for what it's worth, since it's splitting up the inline <span>...</span> element across different messages. Some quick answers in response:

1. Segmentation into a sentence per message is an appropriate unit of length for translators & tools to work at
2. Smaller messages increase the chances for Translation Memory leveraging
3. Allowing Okapi to process allows extraction of translatable attributes inside elements into separate messages, and linking them (ex: a[href], a[title], img[alt])
4. The creation of text units and segments works regardless of markup element nesting depth, and the HTML handler configuration indicates whether an element should be treated as an inline/text-level element (placeholder) or a block level (new message), etc.

I suspect we might be better off keeping MF out of the markup management business in favor of the markup protection business (using something vaguely XLIFF-like for the example):

{Click {+bpt id=a}<a href="mumble">{-bpt}{+bpt id=strong}\{\{#strong\}\}
    {-bpt}here{+ept id=strong}\{\{/strong\}\}{-ept}{+ept id=a}</a>{-ept} to do something}

(Of course that is a lot of work for the developer and fairly unattractive...)

Yes, that would be rather unergonomic.

FWIW, one of my personal takeaways from chatting with @zbraniecki the other day is a better personal appreciation of the subtle problems—bad practices for localization?—that can occur when users author markup like HTML directly into a MF message. I can also see users wanting to insert things like <a href="..." title="..."> and <img src="..." alt="..." /> directly in their messages, and translatable attributes of those elements will probably go overlooked... at least without tooling like Okapi or Fluent. Another example is what if a user authors a message that embeds a block-level HTML element in the message as if it were an inline-level element, not realizing the impedance mismatch? As a hypothetical example, imagine that a user is authoring the equivalent of this as a MF2.0 message:

Please reach out to your local shipping provider from the following list:
<ul>
  <li><a href="..." title="...">DHL</a></li>
  <li><a href="..." title="...">Canada Post</a></li>
</ul>

Even if we set aside how to represent this in MF2.0 syntax, the additional problem here is that you really have a block element (the <ul>) that contains what ought to be separate messages in their own right. What you have is more like an HTML fragment that needs to be broken down further.

So what I'm saying is that I think messages that use markup and authored by hand (ex: by developers) will be problematic and error-prone, unless there is tooling to validate (ex: linters) or to assist (ex: Okapi, Fluent). It feels analogous to the problem with current ICU MessageFormat allowing nested messages (which is tantamount to the rookie l10n mistake of string concatenation) in that these are subtle problems likely to reoccur. If we want to address the problem by strongly encouraging some type of tooling to assist and sidestep problems from handwritten markup, then I think the relative importance of human ergonomics or choosing <element>/{+element} vs. {:markup tag="element"} decreases. This is the reason for me asking the higher level question, "How much benefit do we gain by adding special syntax for markup at all?", although that question doesn't currently doesn't have an issue.

[mihnita]

{:markup type="open" tag="a" ns="html"}Firefox{:markup type="close" tag="a" ns="html"}, which is a {:makup type="open" tag="a" ns="html"}Mozilla{:markup type="close" tag="a" ns="html"} {:markup type="open" tag="loud" ns="ssml"}product{:markup type={"close" tag="loud" ns="ssml"}.

That is not how I see it :-)

This is how I have represented markup-as-placeholders before:

{+a :html type="open"}Firefox{-a :html}, which is a {+a :html}Mozilla{-a :html} {+loud :ssml}product{-loud :ssml}.

With :markup as a function name it means that everything is handled by the same function, and it is hard to modify if that is part of the library.

I will call "the engine" the implementation of the MF2 proper (with parsing, and runtime rendering).
Some functions (the ones in the "official registry") might be provided by the engine.
And the engine can't be modified by the developer using it. That is the case for an implementation if ICU which becomes part of iOS / Windows / Android. And it is true for a MF2 implementation provided by the browser.

With :markup as a custom function it means the dev must provide implementations for ALL supported kinds of markup.
And if :markup is in the standard registry and I want to add my own tag, then I have to register :mihaimarkup, which can't be markup, must be placeholder based.

With them separate I can let the engine handle :html, and I can register my own function for :mihaimarkup, or :ssml.

So the proposal is simply markup = placeholder + open/close attribute.

If you go back to the proposal last year, that is exactly what it says.

I would argue that the two proposals can carry 100% the same information, but the placeholder + function has more flexibility.

A syntax like this is a little less verbose:

{{:img key=icon} {+strong}{$name}{-strong} controls this setting.}

But:

1. assumes html is default, and does not allow for simultaneous markdown to coexist

For example html:sub and ssml:sub

2. Why is a standalone markdown a placeholder, but open/close are completely different concepts (MarkupStart/End)?

If placeholders are good enough for standalone markup, they are also fine for open / close markup (with that extra info about open / close)