Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid 'Too many promises' error #102

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

glamperd
Copy link

@glamperd glamperd commented Apr 4, 2024

See issue #101

A 'Range Error: Too many promises' error is thrown when reading a large file from disk. This error prevents running commands such as 'snarkjs powersoftau prepare phase2' on large files. In such circumstances, the number of promises reaches a limit inherent to node.js.

It is possible to circumvent the error by customising the node.js installation to insert a larger limit. This PR takes the approach of wrapping the promises array in an extra layer of batched promises. Large files can then be read with a native installation of node.js.

I have tested this code by successfully reading chunks of 34gb of data, sufficient to process a 2^28 powers of tau file.

@glamperd glamperd changed the title AVoid 'Too many promises' error Avoid 'Too many promises' error Apr 4, 2024
@OBrezhniev
Copy link
Member

Hi @glamperd!
What is the size of the file you have problem with? 34GB, right?
I've run your unit test with original code on 37GB file and 100TB sparse file with maximum possible buffer size (bytesToRead = 1<<30) and it works fine.
Maybe it's a platform limitation. Looks like you were trying to run it under windows.

@OBrezhniev OBrezhniev self-requested a review May 17, 2024 23:29
@glamperd
Copy link
Author

It was a ~100Gb file (uncompressed ptau with 2^28 points) and it was on Ubuntu. Still, it's possible that environment was a factor, e.g. nodejs version. I was on v18.18.0)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants