Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(buffers): potentially allow configuring max record size, max data file size, etc #13120

Open
Tracked by #13043
tobz opened this issue Jun 13, 2022 · 0 comments
Open
Tracked by #13043
Labels
domain: buffers Anything related to Vector's memory/disk buffers domain: config Anything related to configuring Vector domain: reliability Anything related to Vector's reliability

Comments

@tobz
Copy link
Contributor

tobz commented Jun 13, 2022

As seen in #13102, we have users whose sources may easily ingest enough events in a single go to generate an EventArray that may exceed our configured default limits. While we're going to change the default maximum record size to make the defaults a bit more flexible, we still need a more holistic solution to this problem.

We should investigate the feasibility of allowing values such as the maximum data file size and the maximum record size to be changed by the users via the configuration.

In general, we can (and in some cases, already do) enforce the required invariants between the values, such as making sure the buffer size is greater than or equal the maximum data file size, and so on. We would need to more stringently do so, though, ensuring all values are constrained in whatever way is necessary.

Additionally, and perhaps most important, we would need to investigate if it was safe/possible to allow changing these values when the buffer already existed. In some cases, such as changing the maximum data file size, it would likely not be safe to do so, since much of the logic in the buffer depends on all data files having a uniform size limit, etc... so allowing these changes would possibly require changing the layout of the ledger, etc, essentially encoding the configuration of these values into the buffer in a write-once-read-many fashion.

This would let us at least provide the ability to change the defaults, and it would be up to the user to figure out a deployment strategy to roll out their changes i.e. new hosts that used the new value while old hosts were unchanged and then drained/rotated out of service, etc.

@tobz tobz added domain: config Anything related to configuring Vector domain: buffers Anything related to Vector's memory/disk buffers domain: reliability Anything related to Vector's reliability labels Jun 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: buffers Anything related to Vector's memory/disk buffers domain: config Anything related to configuring Vector domain: reliability Anything related to Vector's reliability
Projects
None yet
Development

No branches or pull requests

1 participant