Skip to content

feat: Spread partitioning for personless overflow #31037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 10, 2025

Conversation

benjackwhite
Copy link
Contributor

Problem

We don't want to keep hot partition issues for overflow if we don't do person processing for some data

Changes

  • Fixes it

Does this work well for both Cloud and self-hosted?

How did you test this code?

@benjackwhite benjackwhite requested a review from pl April 9, 2025 23:50
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Summary

This PR refactors overflow routing logic to improve partitioning for personless events while preserving locality for standard processing. It updates both the overflow assignment in the ingestion consumer and adjusts tests to cover new routing behavior.

  • Updated /plugin-server/src/ingestion/ingestion-consumer.ts to compute a new preserveLocality flag for overflow events based on the SKIP_PERSONS_PROCESSING_BY_TOKEN_DISTINCT_ID setting.
  • Modified /plugin-server/src/ingestion/ingestion-consumer.test.ts tests to assert correct key assignments and routing for both personless and standard events.

2 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile

@benjackwhite benjackwhite merged commit 6af60d4 into master Apr 10, 2025
188 checks passed
@benjackwhite benjackwhite deleted the fix/overflow-routing branch April 10, 2025 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants