formatToParts-like Iterator

[stasm]

Is your feature request related to a problem? Please describe.
Rather than format a message to a string, it's sometimes useful to work with an iterable of formatted parts. This is conceptually similar to NumberFormat.prototype.formatToParts and others. This approach allows greater flexibility on the side of the calling code, and opens up opportunities to design tighter integrations with other libraries.

For instance, in the message Hello, {$userName}!, the userName variable could be a UI widget, e.g. a React component. The code integrating the localization into the React app could then call formatToParts which would yield the React component unstringified, ready to be used in render().

Another example: It is {$currentTime} where $currentTime is a Date object formatted as 18:03. If formatToParts yields it unformatted, the calling code can then call DateTimeFormat.prototype.formatToParts on it and add custom markup to achieve effects like 18:03 (hours in bold).

Describe the solution you'd like
The formatToParts iterator would yield parts after all selection logic runs (in MF terms, that's after plural and select chooses one of the supplied variants) and after the variables are resolved to their runtime values, but before variables are resolved to their values, they are formated to strings, and interpolated into the message.

Describe why your solution should shape the standard
It's an API offering a lot of flexibility to its consumers. The regular format API returning a string can be implemented on top of it, too.

Additional context or examples
Fluent is now considering formatToParts in projectfluent/fluent.js#383 and projectfluent/fluent#273. I expect it to be ready by the end of H1. We see it as a great way of allowing interesting use-cases like component interpolation mentioned above, as well as an alternative approach to handle BiDi isolation (see #28) and to support custom transformation functions for text (great for implementing pseudolocalizations).

[mihnita]

+100

[eemeli]

@stasm How would the "before variables are resolved to their values" bit work? I definitely agree with having parts including the value before its stringified, but what would be the benefit of not already determining the part's original value?

This matters when you consider format-to-parts output together with post-resolution transformations that might enable solutions to #16, #31, #34, and #160. If the parts are emitted before variable resolution, such a transformation could not be applied to them.

[eemeli]

Here's a concrete proposal for the interfaces of the formatted message parts, where the return type of a formatToParts() method or function would be FormattedPart[]:

interface FormattedDynamic<T> {
  type: 'dynamic';
  value: T;
  meta?: FormattedMeta;
  toString(): string;
  valueOf(): T;
}

interface FormattedFallback {
  type: 'fallback';
  value: string;
  meta?: FormattedMeta;
  toString(): string;
  valueOf(): string;
}

interface FormattedLiteral {
  type: 'literal';
  value: string | number;
  meta?: FormattedMeta;
  toString(): string;
  valueOf(): string | number;
}

interface FormattedMessage<T> {
  type: 'message';
  value: FormattedPart<T>[];
  meta?: FormattedMeta;
  toString(): string;
  valueOf(): string;
}

type FormattedMeta = Record<string, string | number | boolean | null>;

type FormattedPart<T = unknown> =
  | FormattedDynamic<T>
  | FormattedFallback
  | FormattedMessage<T>
  | (T extends string | number ? FormattedLiteral : never);

In other words, I'm proposing that we have four different formatted parts:

• literal strings and numbers are directly defined in the source message that's being formatted.
• dynamic values are those returned by custom formatting functions, or from runtime variables. Their types are not defined or forced by the spec.
• message is a resolved message reference, containing a list of message parts.
• fallback is used when errors occur, e.g. a formatting function fails for some reason. Its value should be a syntax-like representation of the expression that caused the error.

The fields of these parts are shared by all, and each has an important role:

• type identifies what sort of part this is
• value is the raw value of the part
• meta, if defined, holds metadata about the value. For example, it may contain an identifier for its gender, or an error_message for a fallback part.
• toString() is used to stringify each part in a locale-appropriate manner. For message this means concatenating its stringified parts; for fallback the value is wrapped as '{value}'.
• valueOf() provides a singular representation of the part's value, for use without the other information included in the part. For dynamic and literal that is the part's raw value, but for message and fallback the toString() output is returned.

In the execution model of the EZ model, these formatted parts may also be used to wrap the arguments of a formatting function, which would allow e.g. for {TITLECASE(-common-term)} to be formatted as the transformed message -common-term, with the whole referenced message as well as each of its parts retaining all of their metadata through the title-casing of their values. A working implementation of this is available: https://github.com/messageformat/messageformat/blob/mf2/packages/messageformat/src/format-message.ts

[longlho]

I'd prefer subclassing dynamic values to be more concrete (see @formatjs/icu-messageformat-parser as a reference. This allows us to write fairly comprehensive linter that analyzes the message and guard against TMS restrictions.
The trickiest one is no-complex-selectors which stems from Smartling's restriction of complexity index cap at 20 (if you flatten all your selectors it should not yield more than 20 unique sentences).

[eemeli]

@longlho Your reference link and at least my understanding of your concerns would indicate that you might be talking about the representation of the source message, rather than this formatted output, where all the selectors, functions etc. have been resolved into a single sequence of formatted parts. Is this so, or have I misunderstood?

The MF2 data model representation of source messages is a separate from this, and its allowance of selectors only at the top level should make it significantly easier to e.g. count selector cases directly.

[longlho]

Ah I misunderstood this then. In that case looks like dynamic is still not enough I believe and would still like more structured data to encompass FormattedDateParts (for example) as @stasm mentioned. I think some primitive types still need to be part of the spec and can be expanded further.

[stasm]

@stasm How would the "before variables are resolved to their values" bit work? I definitely agree with having parts including the value before its stringified, but what would be the benefit of not already determining the part's original value?

Sorry for missing this question back when. Looking at it today, I think I got this wrong. We should resolve the variables references to a runtime value (like the one you proposed in #41 (comment)) and stop there, i.e. yield those runtime values without formatting them to strings.

[stasm]

A naming detail which I think may impact the understanding of the proposed interfaces:

Here's a concrete proposal for the interfaces of the formatted message parts, where the return type of a formatToParts() method or function would be FormattedPart[]:

Would FormattablePart be a better name for this? These objects store the raw value and expose the toString() method which means that the formatting still hasn't happened. FormattedPart implies the opposite, I think.

[stasm]

Actually, let me take a step back. I was under the impression that we'd want to yield unformatted values, but after thinking about this this morning, I'm not so sure anymore.

Given the message: Transferred {NUMBER($fileSize, unit: "megabytes")., would we want to:

1. yield unformatted values:

    {value: "Transferred ", toString(): ..., *toParts(): ...}
    {value: 1.23, toString(): ..., *toParts(): /* calls NumberFormat.formatToParts */}
   

2. yield formatted parts, nested:

    StringPart {value: "Transferred "}
    NumberPart [
        NumberFormatPart { type: 'integer', value: '1' },
        NumberFormatPart { type: 'decimal', value: '.' },
        NumberFormatPart { type: 'fraction', value: '23' },
        NumberFormatPart { type: 'literal', value: ' ' },
        NumberFormatPart { type: 'unit', value: 'MB' }
    ]
   

3. yield formatted parts, flattened:

    StringPart {value: "Transferred "}
    NumberFormatPart { type: 'integer', value: '1' },
    NumberFormatPart { type: 'decimal', value: '.' },
    NumberFormatPart { type: 'fraction', value: '23' },
    NumberFormatPart { type: 'literal', value: ' ' },
    NumberFormatPart { type: 'unit', value: 'MB' }
   

4. some combination of the above, e.g. (2) but also carrying the original raw value?

[eemeli]

@stasm Really good point. And I think it made me change my mind on a few things.

I actually had a decently long reply to this written, but then I realised that my approach to this is premised on

1. Needing/wanting to account for function composition, and the compound values that it effectively requires to be supportable as function arguments due to e.g. message metadata.
2. Enabling lazy reference resolution, which you had also talked about.
3. A desire to align the format-to-parts API with the formatting function argument API.

If instead we skip all of that and require eager resolution for formatting functions args, we really ought to consider alignment with the existing prior art on this as a relatively high priority, i.e. follow what ECMA-402 does. And that to me answers your question: We should go with option 3, formatted & flattened parts, adding something like the source value that formatRangeToParts includes. So something like this:

[
  { type: 'literal', value: 'Transferred ' },
  { type: 'integer', value: '1', source: 1.23 },
  { type: 'decimal', value: '.', source: 1.23 },
  { type: 'fraction', value: '23', source: 1.23 },
  { type: 'literal', value: ' ', source: 1.23 },
  { type: 'unit', value: 'MB', source: 1.23 }
]

Not sure about the exact shape of the source there, mind. The point is, it should allow distinguishing the boundary between one source and the next. I do think that for not-explicitly-formatted non-primitive variable values we ought to have something like { type: 'variable', value: { foo: 'bar' } }.

[stasm]

If instead we skip all of that and require eager resolution for formatting functions args, we really ought to consider alignment with the existing prior art on this as a relatively high priority

Can you explain how the eager vs. resolution for function arguments ties into this? In my mind in both approaches, the parts yielded by formatToParts are a transformation on the function's output. In other words, it doesn't matter when $fileSize is resolved because the part (or parts) depend on the output of NUMBER($fileSize, unit: "megabytes").

[stasm]

And that to me answers your question: We should go with option 3, formatted & flattened parts, adding something like the source value that formatRangeToParts includes.

Message formatting is unique enough that we could justify the nested approach too, kind of like (2) in #41 (comment).

StringPart {value: "Transferred "},
NumberPart {
    value: 1.23,
    parts: [
        NumberFormatPart { type: 'integer', value: '1' },
        NumberFormatPart { type: 'decimal', value: '.' },
        NumberFormatPart { type: 'fraction', value: '23' },
        NumberFormatPart { type: 'literal', value: ' ' },
        NumberFormatPart { type: 'unit', value: 'MB' }
    ]
}

If instead we go for a completely flat output, then I like your idea to use { type: 'literal', value: 'Transferred ' } for the string part.