fs: behaviour of readFile and writeFile with file descriptors

[thefourtheye]

• Version: v11.0.0-pre
• Platform: MacOS, Ubuntu (I have tried in these two operating systems personally)
• Subsystem: fs

This problem has been discussed multiple times without reaching any conclusion. That is why opening a separate issue to discuss.

Documented Expectations

As per our docs

readFile is defined as

Asynchronously reads the entire contents of a file.

and writeFile is defined as

Asynchronously writes data to a file, replacing the file if it already exists.

Current State of readFile & writeFile:

When a regular file path is passed:

The file is either read from the beginning or completely overwritten. This complies with the documentation.

When a file descriptor is passed:

1. readFile
   The file is read from the current position in the file (the current position is maintained by the file descriptor itself).

readFile problem reproduction

'use strict';
const fs = require('fs');

const assert = require('assert');

const buffer = Buffer.alloc(5);
/* Open the file descriptor. */

const fd = fs.openSync(__filename, 'r');
/* Read only the first five characters. */

assert.deepStrictEqual(fs.readSync(fd, buffer, 0, 5), 5);

assert.deepStrictEqual(buffer.toString('UTF-8'), '\'use ');
/* As per the docs, both of these should be the same. */

assert.deepStrictEqual(

fs.readFileSync(fd).toString(),

fs.readFileSync(__filename).toString()

);

2. writeFile
   Writes from the beginning of the file, without replacing the contents of the file. For example, if the existing content in the file is ABCD and the newly written data is 12, then the actual contents of the file after both the writeFiles is 12CD.

writeFile problem reproduction

'use strict';
const fs = require('fs');

const assert = require('assert');
/* Write only five characters. */

fs.writeFileSync('/tmp/test1', 'Hello');

assert.deepStrictEqual(fs.readFileSync('/tmp/test1').toString(), 'Hello');
/* Write something else. */

fs.writeFileSync('/tmp/test1', 'Sun');

assert.deepStrictEqual(fs.readFileSync('/tmp/test1').toString(), 'Sun');
/* Open the file descriptor. */

const fd = fs.openSync('/tmp/test', 'w');
/* Write only five characters. */

assert.deepStrictEqual(fs.writeSync(fd, 'Hello'), 5);

assert.deepStrictEqual(fs.readFileSync('/tmp/test').toString(), 'Hello');
/* Write something else. */

fs.writeFileSync(fd, 'Sun');

assert.deepStrictEqual(fs.readFileSync('/tmp/test').toString(), 'Sun');

These two behaviours deviate from the documented expectations.

Cases for changing the behaviour to work from the beginning

#9671 (It presents the case with readFile)

Cases for keeping the current behaviour

Its intuitive for people to expect read/write from the current position till the end of the file. (For example, read a header from a file and then decide whether to read the rest of the file or not.)

Proposed courses of action

1. Land fs: make writeFile consistent with readFile wrt fd #23709, which makes writeFile also to write from the current file position just like readFile reads from the current file position, and be done with it. (semver-major)
2. Make the readFile and writeFile to work from the beginning of the file. (semver-major)
   a. If the file descriptor is not seekable, an error will be thrown from libuv.
3. Variant of 2, if the file descriptor doesn't correspond to a regular file (basically a non-seekable file descriptor), throw an error. Otherwise, try to work from the beginning of the file. (semver-major)
4. Remove fd support, its just a helper function for a not so oftenly used use-case, and there are valid reasons to support both seeking and non-seeking, so whatever we choose someone will need to write their own wrapper to make functions that work as they expect. Let there be an npm package to do this, or several. (semver-major)
5. Just leave the current behaviour as it is with the doc changes in doc: clarify fd behaviour with {read,write}File #23706.

This is not an exhaustive list. Please feel free to propose more ideas.

PS: This comment by @sam-github summarises my personal expectations from the API.

---

Name	Option 1	Option 2	Option 3	Option 4	Option 5
@thefourtheye		X	X
@mcollina	X			X
@addaleax	X				X
@TimothyGu	X
@mhdawson				X	X
@Trott
@ChALkeR	X + warn *				X
@rvagg	X
@cjihrig	X				X
@joyeecheung	X
@apapirovski	X
@Fishrock123	X
Total	9	1	1	2	4

---

cc @nodejs/fs

Tagging @nodejs/tsc (Might be a bit early, but this has been discussed before. So tagging to get more inputs)

Tagging @seishun @vsemozhetbyt @sam-github @jorangreef @addaleax from the previous discussions.

If needed we can tag the collaborators to get their expectations.

---

Previous discussions on this topic happened in #9671, #10809, #10853, #13572,
#22554

[sam-github]

Another option:

4. Remove fd support, its just a helper function for a not so oftenly used use-case, and there are valid reasons to support both seeking and non-seeking, so whatever we choose someone will need to write their own wrapper to make functions that work as they expect. Let there be an npm package to do this, or several.

[vsemozhetbyt]

See also #22554 for various behaviour of fs.writeFile() variants (sync / callback / promise ). Promise variant adds even more discrepancy: complete rewriting vs positional rewriting vs appending.

[vsemozhetbyt]

As for a use case for fs.writeFile(fd). Say, I have a long process with many iterative tasks. After each task, the script writes a current task identifier to a state file, to save a progress and to be able to restore the state. This identifier should be overwritten, not appended as in a log file, so that it can be easily required not by reading and parsing all the huge log file, but via one small fs.readFile() call. Using a path would have an overhead of a constant file reopening. But we cannot use a file descriptor in this case for now.

[thefourtheye]

@sam-github I included that option in the OP.

@vsemozhetbyt I have included that issue also in the list of references.

---

Let's start recording what we think would be the expected behaviour of readFile & writeFile in the table in OP.

[thefourtheye]

I checked few other languages to see how they handle file descriptors.

Python

• Different functions to deal with file descriptors.
• Positional reads and writes have separate functions.
• No read all or write all functions with file descriptors.

Other languages which I checked were Ruby, Java, C, and C++. They all provide their own abstractions over file descriptors. Using file descriptors are not recommended, so not a lot of fd based functions are available in them.

[Trott]

I think step zero should be to update the documentation to specify the current behavior. If we change that behavior, fine, but there should be no reason to not have correct docs while that is being discussed. Is there already a PR open to update the docs? If not, I'm happy to do it or let @thefourtheye or anyone else do it.

[vsemozhetbyt]

Would it be a bit confusing to state 3 different behaviors in docs? Especially callback/sync fd variant (merging) vs promise fd variant (appending)?

[Trott]

Would it be a bit confusing to state 3 different behaviors in docs? Especially callback/sync fd variant (merging) vs promise fd variant (appending)?

It will be a bit confusing, but hopefully less confusing than the current situation of having the same behavior but incorrectly documented.

[thefourtheye]

@Trott I got a PR up to clarify the behaviour of file descriptors with readFile and writeFile, #23706.

[mcollina]

I'm for deprecating readFile(fd) and writeFile(fd), if it does not break on CITGM. If somebody is deailing with file descriptors, I think they can use the barebone methods read() and write().