Skip to content

Conversation

glamperd
Copy link

@glamperd glamperd commented Apr 4, 2024

See issue #101

A 'Range Error: Too many promises' error is thrown when reading a large file from disk. This error prevents running commands such as 'snarkjs powersoftau prepare phase2' on large files. In such circumstances, the number of promises reaches a limit inherent to node.js.

It is possible to circumvent the error by customising the node.js installation to insert a larger limit. This PR takes the approach of wrapping the promises array in an extra layer of batched promises. Large files can then be read with a native installation of node.js.

I have tested this code by successfully reading chunks of 34gb of data, sufficient to process a 2^28 powers of tau file.

@glamperd glamperd changed the title AVoid 'Too many promises' error Avoid 'Too many promises' error Apr 4, 2024
@OBrezhniev
Copy link
Member

Hi @glamperd!
What is the size of the file you have problem with? 34GB, right?
I've run your unit test with original code on 37GB file and 100TB sparse file with maximum possible buffer size (bytesToRead = 1<<30) and it works fine.
Maybe it's a platform limitation. Looks like you were trying to run it under windows.

@OBrezhniev OBrezhniev self-requested a review May 17, 2024 23:29
@glamperd
Copy link
Author

It was a ~100Gb file (uncompressed ptau with 2^28 points) and it was on Ubuntu. Still, it's possible that environment was a factor, e.g. nodejs version. I was on v18.18.0)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants