Open
Description
opened on Jun 7, 2024
now: trying to produce 100K file and if can't - fallback to 1k files.
Reason:
- now we using kv.BigChunks trick (which reopen rotx every minute) to avoid long-living RoTxs. It works only because blocks are immutable.
- E3 works next way: open rotx, read enough data to create 1 small file (to collector or to compressor), close rotx, build file, index file, merge small files to bigger in background. But blocks data - forcing
small files will be merged in background
Optionally: to make files of size 1K not-compressed. to speedup build, and reduce CPU usage. (probably it's not easy, but wort to take a look).
Activity