Should Combining characters of Category 'Mc' be width 1?

[jquast]

See also:

• Improve handling of combining characters #11
• Wrong width for Hindi on macOS, but correct width on Linux #25
• U+2064 to U+206f are 0 width, but wcwidth returns 1 width #26
• Devanagari's zero-width characters are not accounted for properly #47
• Some emoji have incorrect width #57
• Bengali / Khmer / Gujarati / Odia / Hindi regression? #126

I'm certain I've researched this several times before, and our current measurement of 'Mc' category codes as 0 width is the result of coordination of at least 3 of the issues listed above.

Probably the largest obstacle has been finding 1 or 2 Terminal emulators that correctly display languages that include Mc characters.

[jquast]

UAX #44: Mc (Spacing_Mark): "a spacing combining mark (positive advance width)"

Our history,

• 2015: Improve handling of combining characters #11 72ef7fe @philipc: Fixed bug where Mc was treated as zero-width, On problem: "it classified spacing marks (category Mc) as zero width [...] Characters with a general category of Mc are not included [in zero-width table], and so use the default width of 1."
• 2023: Bugfixes for zero-width characters #91 bbf5bb8 @jquast: Add 'Mc' to zero-width category, Reason: "Bugfixes for zero-width characters" -- Closes nine total issues

Unicode Specification (theoretical): Mc = "positive advance width" -- should be width 1
Terminal Reality (practical): - Indic scripts like Devanagari, Tamil, Kannada render Mc characters as combining marks with zero terminal cells - Issue #47 documented: text alignment broke when Mc was width 1

Our interpretation is INCORRECT per Unicode specs, but CORRECT for matching actual terminal behavior. This library aims to be a general purpose measuring library that (independently) best reflects how most terminals supporting unicode measure width.

I recall on this issue in particular, even if "correctly measured", they may not be legible in most (any?) terminals.

The kinds of languages with issues like this are probably best served with variable-sized fonts and be free from constraints of monospacing.

[jacobsandlund]

I've been investigating this within Ghostty as part of this PR to use my uucode library grapheme break implementation, which actually caused the ucs-detect score for Ghostty to go down.

To research this, I have a branch that measures the visual width of a glyph (the position.x + glyph.size.width), and compares that to the uucode-determined width and logs any mismatches. A mismatch is defined with a tolerance of X pixels, treating the actual width as either including the next cell or not if it spills up to X pixels into it (e.g. a tolerance of 3 px for a 17px-wide cell grid allows a glyph that has a visual width of 20 px to be counted as either one or two cells, but a glyph with visual width 16.999px is treated as one cell).

I tried running ucs-detect but the logs were more stable when I looped over the UDHR files several lines at a time with sleeps in between (see ttylang). I repeated this process both with Mc and Cf as zero width and with no special treatment (East Asian Width with exceptions).

Results

Taking the logs and doing a sort -u on the graphemes results in the following table (the cell grid being 17px wide, so 8.5px being half):

Tolerance	Zero	Spacing	Error Reduction
2.5px	5701	4643	19%
4px	5602	4522	19%
8.5px	5534	4384	21%

(Note: I tried getting the total number of graphemes and got 9659 by logging on every glyph instead of mismatches, though I'm not sure how much I trust that number because there was a lot of log pressure and rendering didn't look as smooth.)

Diff

Taking the git diff --no-index between the zero-width and spacing variants, also including x as visual width (not included in the sort -u counts above), gives the results in this gist: https://gist.github.com/jacobsandlund/3cef918093a7b328d645924f438d4891

For example, here's a dense section (red are errors only present in the zero variant, and green are errors only present in the spacing variant):

-\u091d\u093e → झा    x: 30.69
 \u091d\u0947 → झे    x: 21.30
-\u091f\u093e → टा    x: 24.53
-\u091f\u093e\u0902 → टां    x: 24.53
-\u091f\u093f → टि    x: 24.53
-\u091f\u0940 → टी    x: 24.53
-\u091f\u094b → टो    x: 24.53
 \u091f\u094d\u091c → ट्ज    x: 35.36
-\u091f\u094d\u092f\u093e → ट्या    x: 40.37
-\u0920\u093e → ठा    x: 25.42
-\u0920\u0940 → ठी    x: 25.42
-\u0920\u0948\u0903 → ठैः    x: 22.01
+\u091f\u094d\u091f\u093f → ट्टि    x: 24.53
+\u091f\u094d\u091f\u0940 → ट्टी    x: 24.53
+\u091f\u094d\u0930\u093e → ट्रा    x: 24.53
 \u0921 → ड    x: 17.50
-\u0921\u093c\u093e → ड़ा    x: 26.90

Screenshots

And now a series of screenshots, comparing the spacing variant first, and the zero variant second. This first prints the grapheme followed by a full block, then the grapheme printed twice on the next line:

spacing:

zero:

---

spacing:

zero:

---

spacing:

zero:

---

A case where spacing is wrong while zero is right:

spacing (note that Ghostty caps the width to 2, so to simulate the uucode width of 3 I add a space):

zero:

---

spacing:

zero:

---

Notes

This is with Coretext (Ghostty's default on Mac), though I'd like to try Harfbuzz at some point too.

My font is the Ghostty default "JetBrains Mono", but logging the fallback fonts gives me this (showing the first code point where a fallback font is used):

found codepoint 0x4E16 in fallback face=PingFang SC Regular
found codepoint 0xC138 in fallback face=Arial Unicode MS
found codepoint 0xA78C in fallback face=Menlo Regular
found codepoint 0x712 in fallback face=Noto Sans Syriac Regular Regular
found codepoint 0x1235 in fallback face=Kefa III Regular Regular
found codepoint 0xAA81 in fallback face=Noto Sans Tai Viet Regular
found codepoint 0x1111F in fallback face=Noto Sans Chakma Regular
found codepoint 0x13C2 in fallback face=Geneva
found codepoint 0x1401 in fallback face=Euphemia UCAS
found codepoint 0x791 in fallback face=Noto Sans Thaana Regular
found codepoint 0x1E907 in fallback face=Noto Sans Adlam Regular
found codepoint 0xA2E7 in fallback face=Songti SC Regular
found codepoint 0xA9A5 in fallback face=Noto Sans Javanese Regular
found codepoint 0x92 in fallback face=Songti SC Black
found codepoint 0x179F in fallback face=Khmer MN
found codepoint 0x1A20 in fallback face=Noto Sans Tai Tham
found codepoint 0xD7B in fallback face=Malayalam MN
found codepoint 0x6CF in fallback face=Courier New
found codepoint 0x101C in fallback face=Noto Serif Myanmar Regular
found codepoint 0x529 in fallback face=DIN Condensed Bold
found codepoint 0x1132E in fallback face=Grantha Sangam MN Regular
found codepoint 0xDB8 in fallback face=Sinhala Sangam MN
found codepoint 0x1700 in fallback face=Noto Sans Tagalog Regular
found codepoint 0x2D49 in fallback face=Noto Sans Tifinagh Regular
found codepoint 0xA57A in fallback face=Noto Sans Vai Regular

Finally, you may notice in the gist above that some graphemes log with 0 visual width or multiple times with different visual widths. I'm not too familiar with font shaping, so I'm not positive what's happening there (though my investigation did catch a bug in Ghostty that I'll be PR'ing soon). There are interesting things I caught it doing, like rendering some Tai Tham vowels before the consonants that they follow.

Summary

Handling the many scripts that don't have monospaced fonts in a terminal context is a tricky endeavor. With spacing marks, it's not correct to simply "add one to the width for every spacing mark", but the above shows treating them all as zero width isn't good either. (Note, I even tried "add 0.5 to the width, and take the ceil", but that did worse than the spacing variant above that mostly adds 1.)

So, what then should be done? We could investigate better algorithms that match closer to the existing fonts in these languages. Until then, I think it's better to treat spacing marks according to East Asian Width (usually width 1), and limit the number of overlapping glyphs, like the ones seen above.

[jquast]

Thanks for all of this. It sounds like you suggest that a width of 1 is better "default" for category Mc, even though it is in error for some, the total number of errors are fewer than when it is defined as a width of 0.

With spacing marks, it's not correct to simply "add one to the width for every spacing mark", but the above shows treating them all as zero width isn't good either.

I think i see two classes of issues?

• complex combining rules, "when codepoint is following any of codepoints, width of X, otherwise width of Y". This is common for some kinds of characters, they can be displayed individually, but may sometimes modify the previous, depending on what was displayed previously, one example test case on "combining of HANGUL CHOSEONG and JUNGSEONG"
  • also to note, this library has a "combining browser" but it is made pretty useless because of this rule, it does not know what it should be combined with and combines everything with the letter o. Execute bin/wcwidth-browser.py with argument --combining --wide=1 and --wide=2
• the combined width of complex scripts may be measured as partial cells (pixels) and not discrete cells (0, 1 or 2) as measured in this library.

That has always been my finding, and a difficulty with this specific POSIX-compatible library interface, that it is mainly deciding whether it advances 0, 1, or 2 cells as each next character is processed. There is no allowance for pixels or half-cell calculations by the current API.

But I'm very open to ideas. Do you think it is possible to involve a font engine in our code generation step to create tables to help decide whether a Mc category text is "usually" a width of 0 or 1 when used in its usual way? Or even, to build complex tables, eg. "when codepoint is following any of codepoints, width of X, otherwise width of Y",

If there is anything I can do in ucs-detect to more correctly measure the results I'm happy to help or merge. I could add a --line-sleep-delay, or, maybe I can improve ucs-detect accuracy by mixing in yaml declarations, "expect this result from this series of codepoints", generated by help of a font engine, bypassing wcwidth's naive calculations.