Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fs: add bufferSize option to fs.opendir() #30114

Closed
wants to merge 7 commits into from
Closed

fs: add bufferSize option to fs.opendir() #30114

wants to merge 7 commits into from

Conversation

addaleax
Copy link
Member

Add an option that controls the size of the internal
buffer.

Fixes: #29941

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

@addaleax
fs: add bufferSize option to fs.opendir()
ac9b36b 
Add an option that controls the size of the internal
buffer.

Fixes: nodejs#29941
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. labels Oct 25, 2019
@duffleit
Copy link

What a mob programming session 🙌🏻😀

@jdjuan
Copy link

jdjuan commented Oct 25, 2019

300 people watched the process of this PR during @scriptconf Great job!!!!!

richardlau
richardlau reviewed Oct 25, 2019
View reviewed changes
doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
@drdreo
Copy link

drdreo commented Oct 25, 2019

Can approve that all tests were done

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26194/

@Trott
Copy link
Member

Trott commented Oct 25, 2019

bufferSize will need to be added to test/benchmark/test-benchmark-fs.js

Speaking of which: Needs tests. (Just forgot to git add them maybe? You checked the box for tests and I know it's not like you to not have tests. :-D )

cjihrig
cjihrig reviewed Oct 25, 2019
View reviewed changes
doc/api/fs.md Outdated Show resolved Hide resolved
@addaleax
Copy link
Member Author

bufferSize will need to be added to test/benchmark/test-benchmark-fs.js

Done, thanks!

Speaking of which: Needs tests. (Just forgot to git add them maybe? You checked the box for tests and I know it's not like you to not have tests. :-D )

I checked the box because I added benchmarks that measure the performance impact – I don’t think this is something that we could test for well without digging into internals.

@addaleax addaleax added the semver-minor PRs that contain new features and should be released in the next minor version. label Oct 25, 2019
@richardlau
Copy link
Member

bufferSize will need to be added to test/benchmark/test-benchmark-fs.js

Done, thanks!

Speaking of which: Needs tests. (Just forgot to git add them maybe? You checked the box for tests and I know it's not like you to not have tests. :-D )

I checked the box because I added benchmarks that measure the performance impact – I don’t think this is something that we could test for well without digging into internals.

Test passing non-numbers/negative numbers/zero/non-integers as bufferSize?

@Trott
Copy link
Member

Trott commented Oct 25, 2019 •
edited
Loading

Test passing non-numbers/negative numbers/zero/non-integers as bufferSize?

And maybe one test for 32 and one other reasonable positive integer to confirm that it doesn't throw in those cases?

@addaleax
Copy link
Member Author

@Trott @richardlau I guess, yes – added tests for the error condition/that it works when using a positive integer value.

richardlau
richardlau approved these changes Oct 25, 2019
View reviewed changes
test/parallel/test-fs-opendir.js Outdated Show resolved Hide resolved
@addaleax @richardlau
fixup! fs: add bufferSize option to fs.opendir()
2119b4c 
Co-Authored-By: Richard Lau <riclau@uk.ibm.com>
@addaleax addaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 26, 2019
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26206/

cjihrig
cjihrig approved these changes Oct 26, 2019
View reviewed changes
lib/internal/fs/dir.js Outdated Show resolved Hide resolved
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26218/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26232/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26234/

@addaleax
Copy link
Member Author

Landed in b35181f 🎉

@addaleax addaleax closed this Oct 28, 2019
addaleax added a commit that referenced this pull request Oct 28, 2019
@addaleax
fs: add bufferSize option to fs.opendir()
b35181f 
Add an option that controls the size of the internal
buffer.

Fixes: #29941

PR-URL: #30114
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
@addaleax addaleax deleted the fs-opendir-buffersize branch October 28, 2019 13:35
This was referenced Oct 31, 2019
Closed
Closed
targos pushed a commit that referenced this pull request Nov 5, 2019
@addaleax @targos
fs: add bufferSize option to fs.opendir()
07b5584 
Add an option that controls the size of the internal
buffer.

Fixes: #29941

PR-URL: #30114
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
targos added a commit that referenced this pull request Nov 5, 2019
@targos
2019-11-05, Version 13.1.0 (Current)
64eacd8 
Notable changes:

* cli:
  * Added a new flag (`--trace-uncaught`) that makes Node.js print the
    stack trace at the time of throwing uncaught exceptions, rather than
    at the creation of the `Error` object, if there is any. This is
    disabled by default because it affects GC behavior.
    #30025
* crypto
  * Added `Hash.prototype.copy()` method. It returns a new `Hash` object
    with its internal state cloned from the original one.
    #29910
* dgram
  * Added source-specific multicast support. This adds methods to
    Datagram sockets to support RFC 4607
    (https://tools.ietf.org/html/rfc4607) for IPv4 and IPv6.
    #15735
* fs
  * Added a `bufferSize` option to `fs.opendir()`. It allows to control
    the number of entries that are buffered internally when reading from
    the directory. #30114
* meta
  * Added Chengzhong Wu (https://github.com/legendecas) to
    collaborators. #30115

PR-URL: #30262
@targos targos mentioned this pull request Nov 5, 2019
Merged
targos added a commit that referenced this pull request Nov 5, 2019
@targos
2019-11-05, Version 13.1.0 (Current)
309e1ea 
Notable changes:

* cli:
  * Added a new flag (`--trace-uncaught`) that makes Node.js print the
    stack trace at the time of throwing uncaught exceptions, rather than
    at the creation of the `Error` object, if there is any. This is
    disabled by default because it affects GC behavior.
    #30025
* crypto
  * Added `Hash.prototype.copy()` method. It returns a new `Hash` object
    with its internal state cloned from the original one.
    #29910
* dgram
  * Added source-specific multicast support. This adds methods to
    Datagram sockets to support RFC 4607
    (https://tools.ietf.org/html/rfc4607) for IPv4 and IPv6.
    #15735
* fs
  * Added a `bufferSize` option to `fs.opendir()`. It allows to control
    the number of entries that are buffered internally when reading from
    the directory. #30114
* meta
  * Added Chengzhong Wu (https://github.com/legendecas) to
    collaborators. #30115

PR-URL: #30262
MylesBorins pushed a commit that referenced this pull request Jan 12, 2020
@addaleax @MylesBorins
fs: add bufferSize option to fs.opendir()
ba5683c 
Add an option that controls the size of the internal
buffer.

Fixes: #29941

PR-URL: #30114
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
@targos targos mentioned this pull request Jan 15, 2020
Closed
BethGriggs pushed a commit that referenced this pull request Feb 6, 2020
@addaleax @BethGriggs
fs: add bufferSize option to fs.opendir()
785aa86 
Add an option that controls the size of the internal
buffer.

Fixes: #29941

PR-URL: #30114
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
@MylesBorins MylesBorins mentioned this pull request Feb 8, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@richardlau richardlau richardlau approved these changes

@cjihrig cjihrig cjihrig approved these changes

@Trott Trott Trott approved these changes

@jasnell jasnell jasnell approved these changes

@Fishrock123 Fishrock123 Awaiting requested review from Fishrock123

Assignees
No one assigned
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. semver-minor PRs that contain new features and should be released in the next minor version.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

fs.Dir.read() is very slow
10 participants
@addaleax @duffleit @jdjuan @drdreo @nodejs-github-bot @Trott @richardlau @jasnell @cjihrig @z1c0