Make max checkpoints configurable #22015
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 2 Skipped Deployments
|
nickvikeras
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
|
There is an existing data_limit parameter to better control memory consumption. But I'm ok with making a hard limit configurable |
I tried to use that but found it difficult to tune correctly. I set it very high (8gb) and found it still caused a big drop in throughput. I didn't spend a lot of time debugging it because I found just reducing this hard limit solved the problem. |
nickvikeras
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
bmwill
reviewed
May 2, 2025
| /// | ||
| /// This is read once at startup and cached. Changing the environment variable at runtime will not | ||
| /// have any effect. | ||
| pub static MAX_CHECKPOINTS_IN_PROGRESS: Lazy<usize> = Lazy::new(|| { |
There was a problem hiding this comment.
Can we include this as a part of config vs using an env var?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The 10k limit for in-flight checkpoints is way too high for the analytics indexer. If there is any slowdown in any part of the pipelines we end up having a massive spike from ~2gb to ~60gb memory consumption.
I found we can far better throughput and lower and more predictable memory use if I increase the default batch_size (which is the number of concurrent GCS downloads) from 10 to 1000 (going beyond 1000 just increased memory use without increasing throughput) and limiting the number of checkpoints actually being worked on to ~100. This makes sense to me because:
~<num_cpus>*10compute tasks at the tokio pool is plenty to ensure the CPUs stay busy. Creating more tasks than that is what causes extreme memory explosions when one of the pipelines slows down.Test plan
This code change is already live and after reducing this limit to 100 the backfill has been stable and fast.
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.