Skip to content

block snaps: produce files by 1K blocks #10658

Open
@AskAlexSharov

Description

now: trying to produce 100K file and if can't - fallback to 1k files.

Reason:

  • now we using kv.BigChunks trick (which reopen rotx every minute) to avoid long-living RoTxs. It works only because blocks are immutable.
  • E3 works next way: open rotx, read enough data to create 1 small file (to collector or to compressor), close rotx, build file, index file, merge small files to bigger in background. But blocks data - forcing

small files will be merged in background

Optionally: to make files of size 1K not-compressed. to speedup build, and reduce CPU usage. (probably it's not easy, but wort to take a look).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions