Starting with 10.8.0 I get the last chunk of a previous read stream at the start of the next one

[lll000111]

Platforms: Windows and Linux

Everything is fine as long as I use 10.7.0 or older. The problem starts when I go to 10.8.0 or newer.

I created a test project that reproduces the issue. It

test.tar.gz

• Provides two file, one UTF-8 and one random binary data
• First for the text file: creates a read stream and a write stream to copy the UTF-8 file
• Next creates a read stream and a write stream to copy the BINARY file
• Calculates SHA-256 during the read stream and during the write stream
• For visual inspection shows the beginning of each chunk read from the read stream (converted to text; the text and the binary data are easy to tell apart and the output of the binary is sanitized to not mess up the screen)
• The test code has Flow type annotations &mash; remove them if you don't want them (you can "npm install flow-bin flow-remove-types" for the latest flow version, and the 2nd package installs "flow-node" which is a frontend to node that strips the types when you run JS files through it instead of directly through node)

What you will see is that the write stream for the binary file gets the last chunk from the previous text file's read stream.

The read stream SHA-256 still is good, however, but the write stream's SHA-256 is not. The problem only seems to happen with binary files.

Test project output:

TEXT FILE TEST

  TEXT CHUNK 5.3 Somatosensory Neurons have Receptive FieldsEach subcortical somatose... 65248
  TEXT CHUNK ed the neuron’sreceptive field (Figure 5.3). The neuron’s receptive field ... 65242
  TEXT CHUNK ed/represented. The receptive fields of neuronsinnervating/representing th... 4510

Original text file hash: 9101a84eb2320001628926e3c4decd09eff6680809e95a920db4e18afbcf0201
Read file calculated hash: 9101a84eb2320001628926e3c4decd09eff6680809e95a920db4e18afbcf0201
Write file calculated hash: 9101a84eb2320001628926e3c4decd09eff6680809e95a920db4e18afbcf0201

BINARY FILE TEST

  BINARY CHUNK ed/represented. The receptive fields of neuronsinnervating/representing th.. 61008
  BINARY CHUNK m5^gY5y"$ch86m&KmO}bD,P}r"sySZ6... 65536
  BINARY CHUNK $J|0sd\hsrdh9Tn"?x<0ruZ3KD... 65536
  BINARY CHUNK !>B2}REF7VEdK%Er(*$RuG"U$... 65536
  ...(a few more binary chunks)
  BINARY CHUNK 1NI+`9sHRT)r.V%C&*"#*F_6HKw`... 47536

Original binary file hash: ee1758d957bac3706b6a0ad450ffaeab34d55d5c6d5988a8bef7ce72c8a7db85
Read file calculated hash: ee1758d957bac3706b6a0ad450ffaeab34d55d5c6d5988a8bef7ce72c8a7db85
Write file calculated hash: a9d6ee259f8d6315e5e59db7920d6389d6ebd60618d4a49a8550f8b3e340f06e

[mscdex]

This issue tracker is for reporting bugs in node core and submitting feature requests for node core.

General help questions should be posted to the nodejs/help issue tracker instead.

Issues with third-party modules, npm, or other tools that use node, should be posted to the appropriate issue tracker for that project, unless it can be proven that the issue is in fact with node core and not the module/tool in question.

[lll000111]

THIS IS A NODE BUG!!!

How is this a HELP question????

Or at least tell me why this is supposed to be a "help request", because I'm not actually feeling it, I mean that I'm asking for anyone's help.

[jasnell]

test.tar.gz

Can I ask that you either copy the code into comments here or into a gist. I (and I know others here) are generally unwilling to download an opaque tar ball...

[jasnell]

Also, I assume you mean Node.js 10.8.0 and not 10.0.8?

[lll000111]

EDIT: type annotations removed

Test file 1: streams

'use strict';

const path = require('path');
const fs = require('fs');
const crypto = require('crypto');

module.exports.createReadStream = function (hash, ondata, encoding) {
    return new Promise((resolve, reject) => {
        const stream = fs.createReadStream(
            hash,
            {encoding}
        );

        const cryptoHashObj = crypto.createHash('sha256');

        let buf;

        if (encoding) {
            stream.on('data', data => {
                cryptoHashObj.update(Buffer.from(data, 'utf8'));
                ondata(data);
            });
        } else {
            // Convert the node.js Buffer to an ArrayBuffer and adjust the size if it is not full
            stream.on('data', data => {
                cryptoHashObj.update(data);

                if (data.length < data.buffer.byteLength) {
                    ondata(data.buffer.slice(0, data.length));
                } else {
                    ondata(data.buffer);
                }
            });
        }

        stream.on('error', err => reject(err));
        stream.on('end', () => resolve(cryptoHashObj.digest('hex')));
    });
};


module.exports.createWriteStream = function (filename, encoding) {
    const cryptoHashObj = crypto.createHash('sha256');

    const stream = fs.createWriteStream(
        filename,
        {encoding}
    );

    const write = data => {
        const buf = typeof data === 'string' ?
            Buffer.from(data, encoding) :
            Buffer.from(data);

        cryptoHashObj.update(buf);

        stream.write(buf);
    };

    const end = () => {
        return new Promise((resolve, reject) => {
            const hash = cryptoHashObj.digest('hex');
            stream.once('error', err => reject(err));
            stream.once('finish', () => resolve(hash));
            stream.end();
        });
    };

    stream.once('error', err => console.error('Stream error', err));

    return {
        write,
        end
    };
};

Test file 2: main

 'use strict';

const streams = require('./streams.js');

const UTF8_FILE = '9101a84eb2320001628926e3c4decd09eff6680809e95a920db4e18afbcf0201';
const BINARY_FILE = 'ee1758d957bac3706b6a0ad450ffaeab34d55d5c6d5988a8bef7ce72c8a7db85';

const txtFile = async function () {
    console.log('\nTEXT FILE TEST\n');

    const writeStream = streams.createWriteStream('copy-' + UTF8_FILE, 'utf8');

    const onTxtData = function (data) {
        if (data instanceof ArrayBuffer) {
            throw new Error('What?');
        }

        console.log(
            '    TEXT CHUNK',
            data.substr(0, 75).replace(/\n/g, '') + '...',
            data.length
        );

        writeStream.write(data);
    };

    const readTextFileHash = await streams.createReadStream(UTF8_FILE, onTxtData, 'utf8');
    const writeTextFileHash = await writeStream.end();

    console.log('\nOriginal text file hash:', UTF8_FILE);
    console.log('Read file calculated hash:', readTextFileHash);
    console.log('Write file calculated hash:', writeTextFileHash);
};

const binFile = async function () {
    console.log('\nBINARY FILE TEST\n');

    const writeStream = streams.createWriteStream('copy-' + BINARY_FILE);

    const onBinaryData = function (data) {
        if (typeof data === 'string') {
            throw new Error('What?');
        }

        console.log(
            '    BINARY CHUNK',
            String.fromCharCode.apply(null, new Uint8Array(data)).substr(0, 75).replace(
                /[^A-Za-z 0-9 \.,\?""!@#\$%\^&\*\(\)-_=\+;:<>\/\\\|\}\{\[\]`~]*/g,
                ''
            ) + '...',
            data.byteLength
        );

        writeStream.write(data);
    };

    const readBinaryFileHash = await streams.createReadStream(BINARY_FILE, onBinaryData);
    const writeBinaryFileHash = await writeStream.end();

    console.log('\nOriginal binary file hash:', BINARY_FILE);
    console.log('Read file calculated hash:', readBinaryFileHash);
    console.log('Write file calculated hash:', writeBinaryFileHash);
};

const main = async function () {
    await txtFile();
    await binFile();

    // To run more of those two tests in any sequenc:
    // await txtFile();
    // await txtFile();
    // await txtFile();
    // await binFile();

    console.log('\nYou can also compare ');
};

main().catch(console.error);

[jasnell]

This code is quite strange if all it is doing is copying files and calculating a hash while it happens. With the non-standard syntax here it is going to be impossible to determine if this really is a bug in Node.js, whatever transpiling utility you're using, or in your code. Can I ask you to remove the non-standard syntax from this and see if it has the same problem.

[lll000111]

It is standard javascript, it merely has type annotations. They don't do anything. "Transpiling" merely means removing the types. This is Flow, not TypeScript. I'll remove the types. They are just like comments.

[lll000111]

I updated the code above to remove the types.

The code is the way it is because the actual use case is a) cross platform (React Native, Browser, node.js) to read files on one node, send them via websocket, write them on another. That is why I have a frontend for streams, there is a different one on each platform. Just for the explanation, I think the code is simple enough, just standard stream things, I don't do anything fancy at all. I mean, The node.js part is just read a stream in chunks and write them to a write stream, Most of main.js is just to create a nice example.

And by the way, all that encoding stuff is necessary for the React Native platform which can do binary only using base64 encoding(!). Otherwise I would have only Buffers and no string streams.