Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: make sure to write entire buffer #42434

Closed
wants to merge 13 commits into from
Closed

fs: make sure to write entire buffer #42434

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
4307397
fs: make sure to write entire buffer
ronag Mar 22, 2022
b6b9680
fixup
ronag Mar 22, 2022
a5bf600
fixup
ronag Mar 22, 2022
523314a
fixup
ronag Mar 22, 2022
d4c7a90
fixup
ronag Mar 22, 2022
d0afc94
fixuP
ronag Mar 22, 2022
43f2529
fixuP
ronag Mar 22, 2022
4ff89b4
fixuP
ronag Mar 22, 2022
3869561
fixuP
ronag Mar 22, 2022
ab95708
fixuP
ronag Mar 22, 2022
43d9020
fixuP
ronag Mar 22, 2022
09b7994
fixuP
ronag Mar 22, 2022
be950ef
Update lib/internal/fs/promises.js
ronag Mar 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixuP
  • Loading branch information
@ronag
ronag committed Mar 22, 2022
commit 43f252907fb23663cdc3663e82191a8c476d6459
20 changes: 11 additions & 9 deletions lib/internal/fs/streams.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,8 +387,7 @@ WriteStream.prototype.open = openWriteFs;
WriteStream.prototype._construct = _construct;

function writeAll(data, pos, cb, retries = 0) {
const len = data.length;
this[kFs].write(this.fd, data, 0, len, pos, (er, bytesWritten) => {
this[kFs].write(this.fd, data, 0, data.length, pos, (er, bytesWritten, buffer) => {
if (er?.code === 'EAGAIN') {
er = null;
bytesWritten = 0;
Expand All @@ -398,13 +397,15 @@ function writeAll(data, pos, cb, retries = 0) {
return cb(er);
ronag marked this conversation as resolved.
Show resolved Hide resolved
}
benjamingr marked this conversation as resolved.
Show resolved Hide resolved

ronag marked this conversation as resolved.
Show resolved Hide resolved
this.bytesWritten += bytesWritten;

retries = bytesWritten ? 0 : retries + 1;

if (retries > 5) {
return cb(new Error('writev failed'));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are we trying for 5 times here? Writing a non-zero number of bytes?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment here.

}

this.bytesWritten += bytesWritten;
const len = Byffer.byteLength(data);

if (bytesWritten < len) {
writeAll(data.slice(bytesWritten), pos + bytesWritten, cb, retries);
Expand All @@ -415,11 +416,7 @@ function writeAll(data, pos, cb, retries = 0) {
}

function writevAll(chunks, pos, cb, retries = 0) {
let len = 0;
for (const chunk of chunks) {
len += Buffer.byteLength(chunk);
}
this[kFs].writev(this.fd, chunks, this.pos, (er, bytesWritten) => {
this[kFs].writev(this.fd, chunks, this.pos, (er, bytesWritten, buffers) => {
if (er?.code === 'EAGAIN') {
er = null;
bytesWritten = 0;
Expand All @@ -437,8 +434,13 @@ function writevAll(chunks, pos, cb, retries = 0) {
return cb(new Error('writev failed'));
}

let len = 0;
for (const buf of buffers) {
len += Buffer.byteLength(buf);
}

if (bytesWritten < len) {
writevAll([Buffer.concat(chunks).slice(bytesWritten)], pos + bytesWritten, cb, retries);
writevAll([Buffer.concat(buffers).slice(bytesWritten)], pos + bytesWritten, cb, retries);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend being a bit more subtle here and avoid the massive slowdown. Worst case you are allocating 2x memory to pick one byte. I would recommend just concatenating the buffers that have not been completely written.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cold path though? It should almost never happen. Is it worth optimizing for?

} else {
cb();
}
Expand Down