zstd.Error: Bad threads count - more than 200: 240

[zplizzi]

It seems like zstd.compress(data, level) will fail on a system that has too many cores. My system has 2 64-core CPUs and with 2 threads per, that's 256 virtual cores.

This is especially annoying since the usage of zstd is buried deep in another library so I can't easily pass a threads flag. And ZSTD_NBTHREADS isn't seeming to do anything.

This problem is also mentioned here: Chia-Network/chia-blockchain#14667

[sergey-dryabzhinsky]

Should we just silently set nuber of threads to max limit?

[zplizzi]

I don't know the details of zstd, but i think as long as that wouldn't cause problems that seems fine. Or if there's diminishing returns / downsides to having too many threads, something like min(16, system_max_threads) could be fine too.

[sergey-dryabzhinsky]

No warning, no messages is fine?

[zplizzi]

I think a warning is only necessary if taking the default would cause problems. The user can choose to specify the number of threads if they need more control.

[sergey-dryabzhinsky]

Try latest release 1.5.4.1

[zplizzi]

Thank you, that does seem to fix the issue! Much appreciated :)