Incorrect utf-8 handling on require('file.yaml')

[maxp]

When js-yaml loads long files with multibyte utf-8 symbols sometimes it brokes characters (replaces valid unicode char by two error marks)

yaml = require('js-yaml')
data_broken = require('mydata.yaml')
data_correct = yaml.load( fs.readFileSync('mydata.yaml') )

Probably it happens when utf-8 symbol lays exactly on read buffer boundary and byte reader splits it on two incorrect parts.

[ixti]

Can you please provide a broken YAML example, so I could play with it.
You can either send it to my by e-mail, or fork js-yaml repo and put "failing" YAML file as ./test/issues/data/issue-54.yml

[maxp]

js and yaml:
https://gist.github.com/3997997
https://gist.github.com/3998001

That file filled with cyrillic letter 'у' (two bytes).

The bug appears near 8k buffer boundary on odd aligned utf8 bytes.
Reader splits bytes of the character and produces two broken symbols.

[ixti]

Yes, you're absolutely right. We read stream by 4K chunks, so it gets broken when it reads half of a unicode char.

[ixti]

As we are not using asynchronous work with input streams, there was no point of using fs.openFile under the hood. So I completely dropped support of raw Buffers and file descriptors in favor of plain Strings.