OS X 10.11: system calls return non-normalized unicode strings

[silverwind]

test/sequential/test-chdir.js persistently fails on OS X 10.11 because the output of process.cwd() doesn't match the path we're doing process.chdir() on before.

Here's a reduced test case (npm i mkdirp hexy):

const mkdirp = require('mkdirp');
const hex = require('hexy').hexy;
const dir = __dirname + '/weird \uc3a4\uc3ab\uc3af characters \u00e1\u00e2\u00e3';

mkdirp.sync(dir);
process.chdir(dir);

console.log(hex(new Buffer(dir)));
console.log(hex(new Buffer(process.cwd())));
console.log(dir.length, dir);
console.log(process.cwd().length, process.cwd());

The strings look identical on the terminal, but the bytes differ. Here's the output:

00000000: 2f55 7365 7273 2f73 696c 7665 7277 696e  /Users/silverwin
00000010: 642f 6769 742f 6368 6469 722f 7765 6972  d/git/chdir/weir
00000020: 6420 ec8e a4ec 8eab ec8e af20 6368 6172  d.l.$l.+l./.char
00000030: 6163 7465 7273 20c3 a1c3 a2c3 a3         acters.C!C"C#

00000000: 2f55 7365 7273 2f73 696c 7665 7277 696e  /Users/silverwin
00000010: 642f 6769 742f 6368 6469 722f 7765 6972  d/git/chdir/weir
00000020: 6420 e184 8ae1 85a7 e186 abe1 848a e185  d.a..a.'a.+a..a.
00000030: a7e1 86b2 e184 8ae1 85a7 e186 b620 6368  'a.2a..a.'a.6.ch
00000040: 6172 6163 7465 7273 2061 cc81 61cc 8261  aracters.aL.aL.a
00000050: cc83                                     L.

52 '/Users/silverwind/git/chdir/weird 쎤쎫쎯 characters áâã'
61 '/Users/silverwind/git/chdir/weird 쎤쎫쎯 characters áâã'

cc: @evanlucas

[silverwind]

Just for comparision, could someone post the output of above script on 10.10? Maybe @Fishrock123?

[targos]

output on 10.10.4:

00000000: 2f55 7365 7273 2f6d 7a61 7373 6f2f 7465  /Users/mzasso/te
00000010: 7374 2f77 6569 7264 20ec 8ea4 ec8e abec  st/weird.l.$l.+l
00000020: 8eaf 2063 6861 7261 6374 6572 7320 c3a1  ./.characters.C!
00000030: c3a2 c3a3                                C"C#

00000000: 2f55 7365 7273 2f6d 7a61 7373 6f2f 7465  /Users/mzasso/te
00000010: 7374 2f77 6569 7264 20ec 8ea4 ec8e abec  st/weird.l.$l.+l
00000020: 8eaf 2063 6861 7261 6374 6572 7320 c3a1  ./.characters.C!
00000030: c3a2 c3a3                                C"C#

43 '/Users/mzasso/test/weird 쎤쎫쎯 characters áâã'
43 '/Users/mzasso/test/weird 쎤쎫쎯 characters áâã'

[Fishrock123]

@silverwind Maybe we can check what is at the code points using String.prototype.codePointAt(n) or similar? There are different ways of denoting unicode via code points, and it looks like El Capitan is expanding them. If so, that may actually be an OS bug.

[targos]

@silverwind note that your script has a bug: you should use mkdirp.sync or it fails the first time because the directory doesn't exist yet when process.chdir is executed

[targos]

Perhaps this can help:

const mkdirp = require('mkdirp');
const dir = __dirname + '/weird \uc3a4\uc3ab\uc3af characters \u00e1\u00e2\u00e3';

mkdirp.sync(dir);
process.chdir(dir);

function getChars(str) {
  var chars = [];
  for (var c of str) chars.push(c);
  return chars;
}

var dirC = getChars(dir);
var cwdC = getChars(process.cwd());

console.log(dirC.length);
console.log(cwdC.length);

for (var i = 0; i < dirC.length; i++) {
  if (dirC[i].codePointAt(0) !== cwdC[i].codePointAt(0))
    throw `Different code point at ${i}: ${dirC[i]} - ${cwdC[i]}`;
}

console.log('strings are identical');

On 10.10, strings are identical.

[silverwind]

@targos fixed the mkdirp. Here's your script's output on 10.11:

52
61

/Users/silverwind/git/chdir/codepoints.js:21
    throw `Different code point at ${i}: ${dirC[i]} - ${cwdC[i]}`;
                                                               ^
Different code point at 34: 쎤 - ᄊ

[silverwind]

Either the bug is in uv_cwd or (more likely) in the OS. cc: @saghul

[Fishrock123]

@silverwind could you modify it again to make the output run through String.fromCodePoint()?

Edit: Actually, it would be more helpful do get more out than just one of the conflicts... so log instead of throw also.

[silverwind]

@Fishrock123 not sure if fromCodePoint is of much help. I don't think I can feed bytes into it. For simplicy, here's just a single character compared:

const mkdirp = require('mkdirp');
const hex = require('hexy').hexy;
const dir = __dirname + '/\uc3a4';

mkdirp.sync(dir);
process.chdir(dir);

const a = dir.substring(__dirname.length);
const b = process.cwd().substring(__dirname.length);

console.log(hex(new Buffer(a)));
console.log(hex(new Buffer(b)));
console.log(a.length, a);
console.log(b.length, b);

Output:

00000000: ec8e a4                                  l.$

00000000: e184 8ae1 85a7 e186 ab                   a..a.'a.+

1 '쎤'
3 '쎤'

It goes from 3 to 9 bytes. What kind of unicode encoding is that?

[Fishrock123]

I meant in logging the points ala

for (var i = 0; i < dirC.length; i++) {
  if (dirC[i].codePointAt(0) !== cwdC[i].codePointAt(0))
    throw `Different code point at ${i}: ${dirC[i]} - ${cwdC[i]}`;
}

[silverwind]

@Fishrock123 just comparing the 6 unicode characters (\uc3a4\uc3ab\uc3af\u00e1\u00e2\u00e3):

00000000: ec8e a4ec 8eab ec8e afc3 a1c3 a2c3 a3    l.$l.+l./C!C"C#

00000000: e184 8ae1 85a7 e186 abe1 848a e185 a7e1  a..a.'a.+a..a.'a
00000010: 86b2 e184 8ae1 85a7 e186 b661 cc81 61cc  .2a..a.'a.6aL.aL
00000020: 8261 cc83                                .aL.

6 '쎤쎫쎯áâã'
15 '쎤쎫쎯áâã'
Different code point at 0: 쎤 - ᄊ
Different code point at 1: 쎫 - ᅧ
Different code point at 2: 쎯 - ᆫ
Different code point at 3: á - ᄊ
Different code point at 4: â - ᅧ
Different code point at 5: ã - ᆲ

[targos]

I don't know how chinese works but it's like if OSX decomposed the character into smaller pieces.
Look how the 3 first on the right (ᄊ, ᅧ, ᆫ) can be used together to create the first one on the left (쎤).

What happens with a character like 🚀 '\uD83D\uDE80' ?

[silverwind]

@targos lol, didn't notice it actually was the three parts combined. 🚀 looks fine:

00000000: f09f 9a80                                p...

00000000: f09f 9a80                                p...

2 '🚀'
2 '🚀'

[silverwind]

Noticed the same "assembling" happens on upper ascii characters like umlauts öäü, where the dots are a separate character following the ascii letter. I've seen this before on bugged webfonts, not sure what this mechanism is called.