[css-layout-api] String based inline layout API

[atanassov]

There've been growing number of use cases requiring the ability to perform inline layout on strings that aren't necessary part of any DOM fragment. Consider any application with a complex model using the UA as a render surface through canvas 2d. Current capabilities of canvas a limiting users to fill/strokeText which is quite primitive compared to current richness of html/css - white space handling, font fallback, bidi re-ordering, justification etc. etc.... as well as the ability to have good braking logic and support for hyphenation etc.

So, wouldn't it be great to expose the capabilities of layout (inline in particular) to such context? It seems like the current custom layout APIs are plenty capable of handling inline layout given a constraint space while letting the author decide where the completed fragments go. Can we do the same for string input?

[bfgeek]

This sounds like a pretty good fit for the current API that we have exposed. E.g. you could imaging something like:

const node = new LayoutChild('lots of text');
const f1 = node.generateNextFragment({availableInlineSize: 10});
const f2 = node.generateNextFragment({availableInlineSize: 30}, f1.breakToken);

canvasCtx.fill(f1);
canvasCtx.fill(f2);

There is a few open questions about some requirements:

• When should be exposed in terms of position of glyphs? E.g. presumably some web pages would want to show a caret or similar to "editing" text in a canvas.
• (Any other information needed?)
• How to style runs inside the next, e.g. if a something needed a bold style or similar (canvas obviously provides a way for color, and paint related styles, but would need something for other text features.

E.g. you could imagine an API like:

const node = new LayoutChild('lots of text', [
  {startIndex: 5, endIndex: 10, fontWeight: 900},
  {startIndex: 8, endIndex: 12, fontStyle: 'italic'},
]);

cc/ @kojiishi @fserb

[travisleithead]

Just found this! I can imagine this being awesome for Workers as well (with OffscreenCanvas).

Separately, my team has been working with some customers in one of the "growing number of use cases" (Canvas 2D in fact) to enable this capability, and we just published our ideas as an explainer today: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/Canvas/FormattedText.md

Coincidentally, our explainer and the strawperson sample above are remarkably similar :)

We would love you folks to read it and let us know what you think. Note: it's tailored to work specifically for Canvas, since that seems to be the one place where you can't already leverage the power of CSS/HTML. I'm not opposed to having two separate ways of doing this (one in a LayoutWorklet, one exposed more generally), but we should probably work together on the more general one. Also, if there are important non-canvas use-cases to consider I would be interested in hearing them.

@sushraja-msft

[maltenuhn]

This sounds like a very welcome addition. We have use cases that require precise measurement (dimensions and colours) of rendered styled text on top of various backgrounds, and having a way to achieve that with the ease of CSS would reduce the complexity a lot.

[css-meeting-bot]

The CSS Working Group just discussed String based inline layout API.

The full IRC log of that discussion

<Rossen_> Topic: String based inline layout API
<Rossen_> github:https://github.com//issues/990
<fantasai> Rossen_: More and more desire to be able to perform typographic text layout from Canvas
<Rossen_> https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/Canvas/FormattedText.md
<fantasai> Rossen_: There were some proposals put forward to essentially expose API in the platform that will allow developers to do that
<fantasai> Rossen_: e.g. ^
<fantasai> Rossen_: When I looked at it, and when we discussed it, my initial reaction was, hey, we have Houdini, and Houdini Custom Layout API was intended to handle exactly this
<fantasai> Rossen_: Overall idea, instead of performing layout that is based on a particular subsection of DOM that has text, inline markup, etc.
<fantasai> Rossen_: Instead expose an API that simply, for custom layout or maybe different API, takes as input a string
<fantasai> Rossen_: and does layout into the given space
<fantasai> Rossen_: If Houdini is not able to handle this, we are failing this on some higher level
<fantasai> Rossen_: So talked about it with iank_ and he expanded in the issue why this makes perfect sense and is good fit for the API
<fantasai> Rossen_: Since then see that some additional comments by Travis and ?
<fantasai> Rossen_: from Edge Team
<AmeliaBR> q+
<fantasai> Rossen_: and I wanted to essentially turn this into a question and ask, does it seem like a reasonable API?
<fremy> q+
<fantasai> Rossen_: Is this a good API to add, given intended functionality?
<fantasai> Rossen_: Will it serve the purpose motivating the other proposal?
<fantasai> AmeliaBR: Ian had a question in the issue about, once you've laid out, how do you query back the position of content?
<fantasai> AmeliaBR: If you're going down that road, might look at APIs from SVG
<fantasai> AmeliaBR: Can't promise they'll be all you want, but if can harmonize it's better
<fantasai> AmeliaBR: More specifically on the proposal, like the idea of being able to run layout separately from full CSS layout context
<fantasai> AmeliaBR: And re-using Houdini Layout APIs sounds good
<fantasai> AmeliaBR: I'm concerned about being based on simple text strings
<fantasai> AmeliaBR: As soon as you go beyond the simplest things, need annotations
<fantasai> AmeliaBR: Need info on languages, need markup for bidi, going to need nest it with italic phrase inside regular text phrase, etc.
<iank_> q+
<fantasai> AmeliaBR: So it might be better to keep thinking of styling a DOM fragment
<fantasai> AmeliaBR: but DOM fragment doesn't necessarily need to be attached to the document
<fantasai> AmeliaBR: Because you will need some hierarchy of content
<fantasai> Rossen_: what you're asking for already exists, just extending to handle string only
<Rossen_> ack AmeliaBR
<Rossen_> ack fremy
<fantasai> fremy: One issue of doing with strings is, if you have ? such as text selection
<fantasai> fremy: Doesn't work if you don't have something in the DOM to select
<fantasai> fremy: Don't know the use case for the text
<fantasai> fremy: But selection is something users expect, and problematic if we can't handle selection
<fantasai> fremy: But at the same time, think about what Amelia said
<fantasai> fremy: if you have something in the DOM to represent the string
<fantasai> fremy: [something about spans]
<fantasai> fremy: If in the DOM, you can copy and paste, etc.
<fantasai> fremy: A lot of times you don't want to have everything possible in CSS
<fantasai> fremy: Sounds useful in many cases
<Rossen_> ack iank_
<fantasai> iank_: The linked formatted text explainer is much more well thought out
<fantasai> iank_: and seems pretty good
<fantasai> iank_: Amelia brought up point wrt doing on document fragments
<fantasai> iank_: Issue there is, this API wanted to work inside of workers
<fantasai> iank_: because have the Canvas API
<fantasai> iank_: I don't think needs that much of hierarchy
<fantasai> iank_: styling particular runs is roughly equivalent to structured hierarchy
<sushanth> q+
<fantasai> iank_: One thing to think about there is bidi reordering, though
<fantasai> iank_: To harmonize APIs
<fantasai> iank_: call out some core ? or mixins that can be shared between the APIs
<fantasai> iank_: Once you have a line of text, what info can you query?
<Rossen_> q
<fantasai> iank_: no reason they should be different
<fantasai> ??: Motivation was apps, like photoshop / excel
<fantasai> sushanth: API was trying to give them ability to do multiline text
<astearns> s/??/sushanth/
<fantasai> sushanth: Houdini, they have control over canvas context they'd want text to be rendered?
<fantasai> sushanth: Paint Worklet, Houdini controls context
<fantasai> sushanth: but these cases rendering other things, just want a few lines of text
<fantasai> sushanth: In terms of measuring API, we went towards existing Canvas Text Metrics API
<fantasai> sushanth: exposes same set of things
<fantasai> sushanth: Not the same as SVG, but has precedence on having some APIs
<fantasai> sushanth: Chromium has advances as well
<jfkthame> q+
<fantasai> sushanth: I think last question, selection / background / borders missing from this API
<fantasai> sushanth: with metrics can implement this
<fantasai> sushanth: in JS
<Rossen_> q?
<fantasai> sushanth: Just wanted basic multiline text
<fantasai> jfkthame: I wanted to comment that what I've seen of the text metrics API for canvas there, particularly advances from Chromium, that really does not address the needs of international text well
<fantasai> jfkthame: Does not handle that one character can map to arbitrary number of glyphs and vice versa
<fantasai> jfkthame: the mapping between glyphs and characters can be quite complex, and it doesn't handle that
<Rossen__> q?
<fantasai> AmeliaBR: Hadn't clicked through to explainer previously, and only saw the examples in the issue. Explainer looks like a much more complex model, allows annotations
<fantasai> AmeliaBR: if that's extended to handle i18n issues, that's a good start [wrt bidi and lang?]
<fantasai> AmeliaBR: Might be useful to export a DOM fragment to this format
<fantasai> AmeliaBR: so if you have CSS-styled markup in Canvas Shadow DOM or in your SVG
<fantasai> AmeliaBR: and want to run through this API to get layout, would be a great feature
<Rossen__> q
<sushanth> q+
<fantasai> Rossen__: SVG has had this capability request for the longest time
<fantasai> Rossen__: some such as inkscape that support, no renderers
<fantasai> Rossen__: Want it in Canvas
<fantasai> Rossen__: If for every one of these [sound breaks]
<fantasai> AmeliaBR: He was talking about how SVG has long wanted formatted multiline text, haven't been able to get implemented in browsers, so will be very jealous if Canvas gets it before we do
<astearns> ack sushanth
<fantasai> sushanth: Point about advances in text metrics, it's incomplete
<fantasai> sushanth: flag to get it to wokr
<fantasai> sushanth: Does handle multiple glyphs per characters
<fantasai> sushanth: We augmented with hierarchy to allow advances within ?
<fantasai> sushanth: It addresses multiple chars into single glyph is that it gives you the same advance value
<fantasai> sushanth: They'll all have the same advance value
<fantasai> sushanth: advance = position from the start
<Rossen___> q?
<fantasai> jfkthame: That won't be any different than a zero-width character right?
<AmeliaBR> This is similar to how the SVG API works: multiple characters will match to the same glyph box
<fantasai> sushanth: true
<fantasai> sushanth: Some other issues also?
<fantasai> Rossen__: Main point I wanted to make before network drop...
<fantasai> Rossen__: If we're not looking for ways to harmonize Houdini APIs with rest of platform
<fantasai> Rossen__: failing main use case of exposing ? to other things like Canvas and SVG
<AmeliaBR> s/?/text layout/
<fantasai> Rossen__: As we all know, text seems very benign and easy, when you thinking only about a few ascii glyphs
<fantasai> Rossen__: breaks based on simple logic
<fantasai> Rossen__: Things get increasingly complex
<fantasai> Rossen__: to the point where you're mostly ending up re-implementing layout for CSS
<fantasai> Rossen__: which now intended to be exposed as Houdini APIs
<iank_> q+
<Rossen___> ack fantasai
<Zakim> fantasai, you wanted to ask us to get Myles's opinion before signing off on anything like this
<fremy> q+
<fantasai> fantasai: Just wanted to say, I echo concerns about getting i18n right.
<iank_> q-
<iank_> q+
<fantasai> fantasai: Also want to say, I would be very uncomfortable resolving to add a new major text API without Myles signing off
<fantasai> fremy: Was thinking about use case, the idea is to run this in workers
<fantasai> fremy: because can't ? directly
<fantasai> fremy: So that's why don't want to take something from the DOM and draw it
<AmeliaBR> s/?/access the DOM/
<fantasai> fremy: ...
<fantasai> fremy: HTML content and canvas can automatically update
<fantasai> fremy: what gets drawn on it
<fantasai> fremy: that could get the other things to work
<fantasai> fremy: for ppl using the API
<fantasai> fremy: Draw on the canvas, put this in the DOM, and gets updated
<fantasai> fremy: enables selection, a11y, etc. everything to wokr
<astearns> s/.../describes an export from this API to a DOM fragment/
<fantasai> fremy: That's an idea, another way of looking at it
<fantasai> iank_: Based on the canvas text explainer here, definitely possible to harmonize SVG and layout API with htis
<Rossen___> ack iank_
<Rossen___> ack fremy
<fantasai> iank_: might me helpful to get on videochat and talk about details in detail at some point
<fantasai> Rossen___: As stated in the beginning, wanted to get opinion on whether good approach in general
<fantasai> Rossen___: For the proposed solution put together by ian on that issue, wanted to get feedback from sushanth and others
<fantasai> Rossen___: whether approach worth pursuing
<fantasai> Rossen___: Closing remarks are let's get together and work on this
<sushanth> thank you Ian, happy to follow up. :D
<florian> q+
<Rossen___> ack fantasai
<fantasai> fantasai: bidi, lang-tagging or markup for change in formatting, selection, a11y, character to glyph mapping ...
<fantasai> fantasai: justification? would it be possible?
<Rossen___> q
<fantasai> sushanth: ? has some some text justification, has just left/right/center
<AmeliaBR> s/?/canvas/
<fantasai> fantasai: just wanted to make sure these points don't get lost
<fantasai> florian: Want to echo a bit of the warning Myles gave us last time on this topic
<fantasai> florian: needs to be foolproof, so that if user of API doesn't think about them, they still work
<fantasai> florian: otherwise we make large parts of internet inaccessible or unusable to a lot of the world population
<fantasai> florian: ...
<fantasai> florian: getting the right balance is hard, but we have to do it
<fantasai> iank_: This API is a substantial leap forward from current Canvas APIs
<fantasai> [Note to scribe: insert link to Myle's comments here]