add handling for CJK text around emphasis delimiters #78

QuietMisdreavus · 2025-04-28T20:36:23Z

Resolves rdar://148884965

Problem

Consider the following markdown, taken from commonmark/commonmark-spec#650:

**テスト。**テスト

**テスト**。テスト

**テスト、**テスト

**テスト**、テスト

**テスト？**テスト

**テスト**？テスト

CommonMark, as it is specified today, sees the first line of each pair as "punctuation, emphasis delimiter, text", which forces the asterisks to be considered as a left-flanking delimiter run, unable to close emphasis. This shortcoming exists in GitHub-Flavored Markdown and thus swift-cmark as well, as both of those projects are ultimately based on CommonMark.

The discussion in the above-linked issue has persisted for years as the precise specifications for what should or should not count are defined, but a contributor has created an interim proposed specification erratum as https://github.com/tats-u/markdown-cjk-friendly, specifically their updates in specification.md. While he has some patched Markdown parsers available in his repo, they are all TypeScript based, and thus are not available for swift-cmark as it is written in C.

This PR implements the proposed implementation updates in swift-cmark.

Implementation

Rather than perform all the necessary calculations of what is and is not a "CJK character" according to the proposed specification in C, this PR takes a note from the author and pre-computes the list of code points in a Swift script, which is then run to generate a cjk-ranges.inc file used from swift-cmark itself. This script uses a handful of Unicode data files, which i have also included in this PR alongside the existing Case Folding data file inherited from CommonMark.

Copying in part of the proposed specification:

A CJK character is a character (Unicode code point) that meets at least one of the following criteria:

Meets both of the following criteria:

UAX #11 East Asian Width category is either W, F, or H

Not in RGI emoji set (i.e. is not fully-qualified emoji) defined in UTS #51 Unicode Emoji

UAX #24 Unicode Script Property is Hangul

While the use of "fully-qualified emoji" is a bit problematic for code that only processes a single code point at a time (that definition includes multi-code-point sequences like skin tone variants and regional flags), the solution in the adapted code focuses on those code points that are themselves valid emoji sequences. This is enough to filter out any confounding code points that would otherwise mess with the calculation, based on some rough smoke testing i performed.

Once the calculated "CJK character" ranges are saved, the remainder of the PR is relatively straightforward: Handle a few more ranges of code points mentioned in the proposed spec, then update the calculation for left- and right-flanking delimiters and you're done.

Acknowledgements

I would like to thank @tats-u from the bottom of my heart for having put in the work to develop this proposed specification and all the supporting code present in his repository. It feels to me like a lot of work has been put into it, and i really appreciate it.

rdar://148884965

tools/make_cjk_ranges.swift

patshaughnessy

Great work 👏 👏

I have various nitpicks about the Swift tool you added (which is a brilliant idea!) and also a few questions about the core algorithm itself.

tools/make_cjk_ranges.swift

patshaughnessy · 2025-05-22T20:49:09Z

tools/make_cjk_ranges.swift

+    return (cjkRanges, nonCjkRanges)
+}
+
+/// Returns the ranges of code points with a Script property of Hangul.


Are we sure Hangul is the only East Asian script in this file? I see various Indian scripts and also Tibetan. Also Hiragana, Katana and various other Asian scripts.

I'm sure this is correct; but maybe add a comment explaining why these other scripts are not needed here. Maybe their code points are included in EastAsianWidth.txt?

Chinese and Japanese are covered by EAW in (Wide, Full, Half).
There are some Hangul characters whose EAW is Neutral.

I see various Indian scripts and also Tibetan.

I've not observed actual complaints by speakers of those languages. Thai might be suffered from this CommonMark specification bug, but I've never heard complaints.

tools/make_cjk_ranges.swift

patshaughnessy · 2025-05-22T21:09:19Z

src/utf8.c

@@ -315,3 +315,20 @@ int cmark_utf8proc_is_punctuation(int32_t uc) {
      uc == 92917 || (uc >= 92983 && uc <= 92987) || uc == 92996 ||
      uc == 113823);
 }
+
+int cmark_utf8proc_is_cjk_character(int32_t uc) {


I'd have the Swift tool produce the function definition also; this would make it more clear where the parameter uc comes from, and where the generated code goes to.

Probably "Unicode Code Point".

I disagree with having the Swift code generate the function definition, since it makes it more difficult to ensure that the function signature matches the header's definition. It also allows me to change the name of the function later without having to change the generation code and rerun it.

uc was chosen to match the other UTF-8 functions in cmark; as @tats-u mentioned, the etymology is likely "Unicode Codepoint".

patshaughnessy · 2025-05-22T21:18:04Z

src/inlines.c

+      before_before_char = 10;
+    } else {
+      before_before_char_pos = before_char_pos - 1;
+      // walk back to the beginning of the previous UTF-8 sequence again:


Is this entirely the same as the previous walk-back code? Should we extract a function?

From a quick search, it looks like there's one other place that performs reverse UTF-8 iteration (cmark_inline_parser_scan_delimiters, also in inlines.c). It might be worth factoring out a separate reverse-iteration function, but i would hesitate to do so, just to keep the overall diff from GFM smaller.

patshaughnessy · 2025-05-22T21:26:54Z

src/inlines.c

                   cmark_utf8proc_is_space(before_char) ||
-                   cmark_utf8proc_is_punctuation(before_char));
+                   cmark_utf8proc_is_non_cjk_punctuation_character(before_char) ||
+                   cmark_utf8proc_is_cjk_character(before_char) ||


Question about the core algorithm behind this: Why do we consider the delimiter to be left flanking if the preceding character is any cjk character? Should we distinguish between cjk punctuation and other cjk characters?

Put a different way: Before this change we consider the delimiter to be left flanking if:

After is not whitespace AND

After is not punctuation OR Before is whitespace OR Before is punctuation

This PR expands on the "Before is whitespace OR Before is punctuation" logic, additionally allowing the before character to be:

a CJK character

an Ideographic Variation Selector

a Standard Variation Selector

But this seems very generic: Any preceding CJK character would match. Should we check if the CJK character is punctuation? Similar to the original logic?

Before: After is NOT punctuation OR Before is (whitespace OR punctuation)
↓
The strategy: After is (NOT punctuation OR CJK) OR Before is (whitespace OR punctuation OR CJK)
↓
After is NOT (punctuation AND NOT CJK) OR Before is (whitespace OR ((punctuation AND NOT CJK) OR (punctuation AND CJK)) OR CJK)
↓
After is NOT (punctuation AND NOT CJK) OR Before is (whitespace OR (punctuation and NOT CJK) OR ((punctuation AND CJK) OR CJK))
↓
After is NOT (punctuation AND NOT CJK) OR Before is (whitespace OR (punctuation and NOT CJK) OR CJK)

"punctuation AND CJK" vanishes by this expression transformation.

patshaughnessy · 2025-05-22T23:07:41Z

src/inlines.c

+  //   - followed by one of the following:
+  //     - whitespace
+  //     - a non-CJK punctuation character
+  //     - a CJK character


Same question here: Should we refine the check for any CJK character to instead look for CJK punctuation characters only?

Also: Why don't we include Ideographic Variation Selectors and Standard Variation Selector like we do for the left flanking logic?

Why don't we include Ideographic Variation Selectors and Standard Variation Selector like we do for the left flanking logic?

People other than penetration testers never input a (non-han character → ideographic variation sequence) sequence. 99.9+% of ideographic variation selectors follow a han character. This is just an optimization.
"Standard Variation Selectors" only follow non-variation-selector characters.

Addition: These variation selectors won't come to the first code point of a grapheme cluster. i.e. They won't follow * or _.

To expand a little bit: Variation Selector characters are meant to go after another code point, the way that skin-tone selectors go after a base emoji. For that not to be true, a file would need to either be corrupted or specially created.

The one special code point i would expect to see right after an asterisk is the Emoji Variation Selector, which (IMO) should render the asterisk incapable of creating emphasis. As far as i know, though, that discussion hasn't happened as part of the greater "CJK emphasis" thread.

patshaughnessy · 2025-05-22T23:09:24Z

I've also tested this with a large CJK markdown document, and it worked perfectly as far as I could tell (I don't speak any of the target languages, however.)

tats-u

I have forgotten to enable Discussion in my repo. If you have additional questions about the specs, you may want to post them there:

https://github.com/tats-u/markdown-cjk-friendly/discussions

Note: the following comments are as answers for questions on the specs.

tats-u · 2025-05-25T07:45:57Z

src/utf8.c

@@ -315,3 +315,20 @@ int cmark_utf8proc_is_punctuation(int32_t uc) {
      uc == 92917 || (uc >= 92983 && uc <= 92987) || uc == 92996 ||
      uc == 113823);
 }
+
+int cmark_utf8proc_is_cjk_character(int32_t uc) {


Probably "Unicode Code Point".

tats-u · 2025-05-25T07:46:04Z

src/inlines.c

                   cmark_utf8proc_is_space(before_char) ||
-                   cmark_utf8proc_is_punctuation(before_char));
+                   cmark_utf8proc_is_non_cjk_punctuation_character(before_char) ||
+                   cmark_utf8proc_is_cjk_character(before_char) ||


Before: After is NOT punctuation OR Before is (whitespace OR punctuation)
↓
The strategy: After is (NOT punctuation OR CJK) OR Before is (whitespace OR punctuation OR CJK)
↓
After is NOT (punctuation AND NOT CJK) OR Before is (whitespace OR ((punctuation AND NOT CJK) OR (punctuation AND CJK)) OR CJK)
↓
After is NOT (punctuation AND NOT CJK) OR Before is (whitespace OR (punctuation and NOT CJK) OR ((punctuation AND CJK) OR CJK))
↓
After is NOT (punctuation AND NOT CJK) OR Before is (whitespace OR (punctuation and NOT CJK) OR CJK)

"punctuation AND CJK" vanishes by this expression transformation.

tats-u · 2025-05-25T07:49:26Z

src/inlines.c

+  //   - followed by one of the following:
+  //     - whitespace
+  //     - a non-CJK punctuation character
+  //     - a CJK character


Addition: These variation selectors won't come to the first code point of a grapheme cluster. i.e. They won't follow * or _.

QuietMisdreavus · 2025-06-02T23:52:23Z

I pushed up some commits to respond to some of @patshaughnessy's review comments, and to update the algorithm to match the most recent release of markdown-cjk-friendly, and its use of punctuation variation sequences. Hopefully the code should be a bit easier to understand now. 😅

tats-u · 2025-06-05T23:29:29Z

You should remove test cases that are still correctly recognized as intended by pure CommonMark/GFM parsers.

A**BCD**E

We have only to care about whether B or D are punctuation or not, and whether A, B, D, or E are CJK or not.

tats-u · 2025-07-03T10:19:08Z

api_test/main.c

+    "テスト✌🏻**テスト？**テスト\n"
+    "\n"
+    "テスト✌🏻**テスト**？テスト\n"
+    "\n"
+    "テスト🇯🇵**テスト？**テスト\n"
+    "\n"
+    "テスト🇯🇵**テスト**？テスト\n"
+    "\n"
+    "テスト🏴󠁧󠁢󠁳󠁣󠁴󠁿**テスト？**テスト\n"
+    "\n"
+    "テスト🏴󠁧󠁢󠁳󠁣󠁴󠁿**テスト**？テスト\n"
+    "\n"
+    "テスト*️⃣**テスト？**テスト\n"
+    "\n"
+    "テスト*️⃣**テスト**？テスト\n"
+    "\n"
+    "テスト©️**テスト？**テスト\n"
+    "\n"
+    "テスト©️**テスト**？テスト\n"
+    "\n"
+    "テスト©**テスト？**テスト\n"
+    "\n"
+    "テスト©**テスト**？テスト\n"
+    "\n"
+    "テスト⌛**テスト？**テスト\n"
+    "\n"
+    "テスト⌛**テスト**？テスト\n";


These emojis and symbols have no effect. The following is better:

テスト**テスト？**test テスト**✌🏻テスト**テストテスト**テスト?**テスト test**「テスト」**テスト

add handling for CJK text around emphasis delimiters

1a3c10e

rdar://148884965

QuietMisdreavus requested review from patshaughnessy, d-ronnqvist, franklinsch and mayaepps April 28, 2025 20:36

tats-u reviewed Apr 29, 2025

View reviewed changes

tools/make_cjk_ranges.swift Outdated Show resolved Hide resolved

tats-u mentioned this pull request Apr 29, 2025

Emphasis with CJK punctuation commonmark/commonmark-spec#650

Open

review: add Unicode version comment to make_cjk_ranges.swift

c338b7b

patshaughnessy approved these changes May 22, 2025

View reviewed changes

tats-u reviewed May 25, 2025

View reviewed changes

QuietMisdreavus added 6 commits June 2, 2025 13:27

refactor: split out the data directory folder reference

0355f43

review: add comment referencing EAW values

6119151

review: remove unnecessary guard

329565e

refactor: pull out common regex components into functions

5a1122e

review: add comments about text/emoji presentation selectors

2ec4098

add support for tats-u/markdown-cjk-friendly#3

fc19fba

tats-u reviewed Jul 3, 2025

View reviewed changes

add handling for CJK text around emphasis delimiters #78

Are you sure you want to change the base?

add handling for CJK text around emphasis delimiters #78

Uh oh!

Conversation

QuietMisdreavus commented Apr 28, 2025

Problem

Implementation

Acknowledgements

Uh oh!

Uh oh!

patshaughnessy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tats-u May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tats-u May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patshaughnessy commented May 22, 2025

Uh oh!

tats-u left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

QuietMisdreavus commented Jun 2, 2025

Uh oh!

tats-u commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tats-u May 23, 2025 •

edited

Loading

tats-u May 23, 2025 •

edited

Loading

tats-u left a comment •

edited

Loading

tats-u commented Jun 5, 2025 •

edited

Loading