Encourage effective use of Unicode

[sunfishcode]

Treating Unicode strings as arrays often leads to bugs where code processes text in some languages correctly but not others. In JavaScript, it's surprising that "🤦🏼‍♂️".length == 7, and the advice to programmers often is: you usually don't want to look at .length, because it isn't reliably what end users think of as characters, it isn't reliably the number of codepoints, and it isn't reliably related to the display width of the string.

Similarly, functions names borrowed from JS using the term "Char", such as fromCharCode, are confusing to programmers coming from non-JS languages, since code units aren't always characters.

So, what if AssemblyScript moved functions which work in terms of the underlying code-unit concept, such as charCodeAt, into a String.JS namespace, similar to the String.UTF16 namespace? They'd all be available, and easily accessible. But, they'd be visually distinguished from the other string functions, making it clear where code-unit assumptions are being made. It would also leave more conceptual room in the base String namespace for new features in the future.

Another effect of the name String.JS could be to signal to programmers that these functions won't necessarily always be optimal or natural in non-JS embeddings of Wasm, which may give AssemblyScript as a language more implementation flexibility in non-JS environments.

All that said, I don't know where AssemblyScript stands on standard library API stability at this time. If breaking changes are out of scope, perhaps some of the above goals could at least be advanced through documentation.

[willemneal]

I agree. One way to avoid the braking change is to deprecate the current string API that produces a message outlining the move to String.JS and a link to documentation of why.

Also, what is the preferred way of getting the proper length in JS?

[MaxGraey]

In JavaScript, it's surprising that "🤦🏼‍♂️".length == 7

Honestly this is not only problem of JavaScript. But Rust, Swift, Java and all other languages which measure string length in at least code points instead graphemes. All languages exposed some new api for iterate over graphemes. For example Rust:

use unicode_segmentation::UnicodeSegmentation;

fn main() {
    for g in "नमस्ते्".graphemes(true) { // hindi
        println!("grapheme - {}", g);
    }
}

JavaScript will do this (with Intl.Segmenter which support only in Chrome Canary and FF nightly for now):

const segmenter = new Intl.Segmenter("hi", { granularity: "grapheme" });
for (const { segment: g } of segmenter.segment("नमस्ते्")) {
  console.log("grapheme: ", g);
}

[MaxGraey]

I agree. One way to avoid the braking change is to deprecate the current string API that produces a message outlining the move to String.JS and a link to documentation of why.

I don't think we should rework existing String or expose something new which fully unicode aware (Str for example). This will blowup runtime significantally. See comment author of swiftwasm which thinking of switch back to weekness UTF-16 variant of strings due to runtime of Swift is up to 5-6 mb just for hello world.

Also in most of cases we don't really care about fully compatible UTF strings. But if you really need it you could include something like Intl.Segmenter.

[MaxGraey]

By the way, no one language supports all special cases of unicode case mapping. Just most popular like Greek Final Sigma (which context dependent): rust-lang/rust#26035

we also support this:
#1113

But SpecialCasing.txt contain a lot more special complex cases which nobody handle (expect languages which support full ICU library which increase size of their runtime really significant)

[MaxGraey]

Unicode standard is really mess currently, it has a lot of redundant glyphs, and really large amount of special cases which constantly grow from version to version.

[MaxGraey]

@sunfishcode

Similarly, functions names borrowed from JS using the term "Char", such as fromCharCode, are confusing to programmers coming from non-JS languages, since code units aren't always characters.

Since ES6 JS also support code points via Array.from, str.pointAt, str.codePointAt, String.fromCodePoint methods. other methods like toUpperCase as unicode aware except split. Simple example:

function reverseStringNaive(str) {
  return str.split("").reverse().join("");
}

function reverseStringUnicodeAware(str) {
  return Array.from(str).reverse().join("");
  // or [...str].reverse().join("");
}

console.log(reverseStringNaive("foo 𝌆 bar")); 
// will print:
//> rab �� oof
console.log(reverseStringUnicodeAware("foo 𝌆 bar"));
// will print:
//> rab 𝌆 oof

Also, what is the preferred way of getting the proper length in JS?

@willemneal

// calc length in O(1)
console.log("🤦🏼‍♂️".length);  // "7"

// calc length in O(N)
console.log([..."🤦🏼‍♂️"].length); // "5" and now it align with Rust's "🤦🏼‍♂️".chars().count() result

iterate by code point (unicode aware):

for (const c of "foo 𝌆 bar") {
  console.log(c);
}

So JS/TS/AS contain all instruments for handling with unicode strings and even graphemes.
But It don't need in 95% of cases)

[MaxGraey]

Here's what the official ICU documentation says about linrary size:

ICU includes a standard library of data that is about 16 MB in size. Most of this consists of conversion tables and locale information. The data itself is normally placed into a single shared library.

Update: as of ICU 64, the standard data library is over 20 MB in size. We have introduced a new tool, the ICU Data Build Tool, to replace the makefiles explained below and give you more control over what goes into your ICU locale data file.

So some languages (like Swift) use or small_icu mode of ICU or use own custom implementations (Go, Rust, C#, C++) which contain only really necessary unicode property tables compressed as static tries (three staged indirect lookups). We also follow this way.

[MaxGraey]

Regarding string interfaces. What we can do for make it mode safer for users? That's good question. I guess we could declare some methods as deprecate and non recommend for usage (like charCodeAt, String.fromCharCode, split and others). Typescript's definition file supporting /// @deprecate directive comment now. We have own definitions for stdlib so this possible. Secondary we could ban this methods with alternative suggestions in --pedantic mode. @dcodeIO wdyt?

[dcodeIO]

I am not sure about deprecating these, like if one wants to deal with a WTF-16 string, then these are still the way to go I think? For instance, we use them for small strings in the loader because that's faster than piping through a TextDecoder. Of course we can add more details to the documentation, with a neutral link for everyone interested to learn more?

[MaxGraey]

I meant deprecate some string methods inside AssemblyScript in future when Array.from and iterators landed

[dcodeIO]

I am not so sure about that, unless JS itself officially discourages their use. Otherwise we are just creating unnecessary barriers, aren't we?

[MaxGraey]

How about deprecate it only for --pedantic mode?

[MaxGraey]

Btw Rust also required special counting api for code points (aka CCs/Character Codes):

println!("utf8 units (bytes): {}", "🤦🏼‍♂️".len()); // 17  ->  17 bytes
println!("code points: {}", "🤦🏼‍♂️".chars().count()); // 5

JS:

console.log("utf16 units (ushorts):", "🤦🏼‍♂️".length); // 7    "🤦🏼‍♂️".length * 2  ->  14 bytes
console.log("code points:", [..."🤦🏼‍♂️"].length); // 5

So Rust's len() and JS .length just retrieve encoded length in different unit spaces (like ft and m) which could simply converted to bytes and this makes sense. If you need count of character codes you should call different method, if you want count it as visible/writing units (graphemes) you should deal with totally another api.

[dcodeIO]

Really not sure. If the broader ecosystem goes that route, perhaps a diagnostic message about a potential pitfall, but I don't see a reason for asc to spearhead something like that. At this point in time, I'd expect that most users would complain about it.