fs: cannot interact with invalid UTF-16 filenames on Windows, even with Buffers

Version: 10.12.0
Platform: Windows 10 64-bit
Subsystem: fs

PR #5616 gave us support for Buffer paths in all fs methods, primarily to allow interacting with files of unknown or invalid file encoding. This helps on UNIX/Linux where filenames are technically just strings of bytes and do not necessarily represent a valid UTF-8 string.

Similarly, on Windows, filenames are just arrays of wchars, and do not necessarily represent a valid UTF-16 string, however the current { encoding: 'buffer' } variety of fs methods do not properly handle this case. Instead, the Buffers that are returned are UTF-8 representations of (potentially losslessly / incorrectly) decoded UTF-16 filenames. Similarly, it's not possible to pass as input Buffers that represent the raw UTF-16 bytes. This leads to the possibility of files that Node can't interact with at all.

Consider the following code that makes a file that doesn't have a proper UTF-16 name. The created file can be seen and interacted with using Windows Explorer and Notepad without issue.

#include "stdafx.h"
#include <iostream>
#include <windows.h>
#include <string>
using namespace std;

int main()
{
       // Junk surrogate pair
   const wchar_t *filename = L"hi\xD801\x0037";
   HANDLE hfile = CreateFileW(filename, GENERIC_READ, 0, NULL, CREATE_NEW, FILE_ATTRIBUTE_NORMAL, NULL);
   return 0;
}

Then, running the following Node code in the same directory shows that the file cannot be accessed:

const fs = require('fs');

const bufs = fs.readdirSync('.\\', { encoding: 'buffer' });
for (const buf of bufs) {
    try {
        const stat = fs.statSync(buf);
        console.log('successfully got stats of: ' + buf.toString('utf8'));
    } catch (err) {
        console.log('error getting stats of: ' + buf.toString('utf8'));
    }
}

The above code produces the following output when run in the same dir as the invalid UTF-16 file:

error getting stats of: hi�7
successfully got stats of: test.js

Refs:
#5616
rust-lang/rust#12056
jprichardson/node-fs-extra#612

I think the issue here is that libuv attempts automatic UTF-8 → UTF-16 conversion for Windows file paths… /cc @nodejs/libuv

Correct. From http://docs.libuv.org/en/v1.x/fs.html:

Note: On Windows uv_fs_* functions use utf-8 encoding.

You feed it UTF-8 and libuv takes care of converting it to/from WCHAR.

Perhaps Node.js could override WideCharToMultiByte and MultiByteToWideChar to make libuv use WTF-8 instead of UTF-8?

@bnoordhuis

@bnoordhuis @seishun @addaleax Should this be labeled blocked as waiting for an upstream fix in libuv? Or help wanted? Or closed as not-a-bug? Or something else?

@Trott

@Trott This might be possible to fix without changes in libuv, but I would like some input on my idea before I proceed with investigation.

@vtjnash

It seems to me this might be already fixed in libuv as libuv/libuv#2970 landed. Is this correct @vtjnash ? If that's the case the next libuv release will have it and when it lands in nodejs this will be fixed.

Yes. Might need testing, but that is the expectation as long as nodejs don't have a strictly-validating utf8 check in the way

I believe this is fixed now. Closing but holler if it should be reopened.

Navigation Menu

Repository navigation

fs: cannot interact with invalid UTF-16 filenames on Windows, even with Buffers #23735

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fs: cannot interact with invalid UTF-16 filenames on Windows, even with Buffers #23735

Description

Activity

addaleax commented on Oct 18, 2018

bnoordhuis commented on Oct 20, 2018

seishun commented on Nov 11, 2018

Trott commented on Nov 21, 2018

seishun commented on Jan 1, 2019

santigimeno commented on May 25, 2023

vtjnash commented on May 25, 2023

bnoordhuis commented on Aug 6, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions