Protect against aggregating problematically large documents (part 1) - #3804
Open
SethSmucker wants to merge 3 commits into
Open
Protect against aggregating problematically large documents (part 1)#3804SethSmucker wants to merge 3 commits into
SethSmucker wants to merge 3 commits into
Conversation
Adds an entry-count limit on document aggregation. When a document reaches doc.agg.max.entries entries (default -1, disabled), the new EventDataQueryEntryLimitFilter rejects further entries, adds a synthetic INCOMPLETE_DOCUMENT marker field so the truncation is visible in results, and issues a seek past the remainder of the document so the excess entries are not even read. The filter decorates whatever event evaluation filter the query already uses; standard event queries only (TLD/ancestor/parent iterators supply their own filters and are out of scope here).
apmoriarty
reviewed
Aug 5, 2026
|
|
||
| if (config.isSeekingEventAggregation()) { | ||
| addOption(cfg, QueryOptions.SEEKING_EVENT_AGGREGATION, String.valueOf(config.isSeekingEventAggregation()), false); | ||
| addOption(cfg, QueryOptions.DOC_AGGREGATION_MAX_ENTRIES, String.valueOf(config.getDocAggregationMaxEntries()), false); |
Collaborator
There was a problem hiding this comment.
This option should be independent of a seeking aggregation.
Collaborator
|
I see this is marked as part one. Please include an integration test that uses AbstractQueryTest and AbstractIngest to demonstrate the event short circuit behavior. It's fine if the same large event is ingested repeatedly with different event ids. Tests should demonstrate queries that hit entirely within the aggregation window, and also include queries with terms that fall outside the aggregation window (and then document results not coming back, or document an exception to the max size threshold). |
…n tests Review feedback on the large document protection PR: the option was only sent to the iterator when seeking event aggregation was enabled. Hoist it into its own guard so the entry limit applies independently. Add LargeDocumentProtectionIT covering truncation and marker behavior with seeking on and off, documents under the limit, and query terms that fall outside the aggregation window (both indexed and event-only).
SethSmucker
force-pushed
the
3656-large-doc-protection
branch
from
August 5, 2026 20:14
f8e6360 to
b31bb69
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3656
Document aggregation in KeyToDocumentData.collectDocumentAttributes is unbounded. An exceptionally large document gets buffered in its entirety and can exhaust tserver memory, and the existing doc.agg.threshold stopwatch only logs a warning after the fact. I added a doc.agg.max.entries option (default -1, disabled) and an EventDataQueryEntryLimitFilter that wraps whatever event evaluation filter the query already uses. Once a document hits the limit, the filter rejects the remaining entries, adds a synthetic INCOMPLETE_DOCUMENT marker field so the truncated document is identifiable in results, and seeks past the rest of the document so the excess entries are not read at all.
This is the first tier of the issue and covers standard event queries only. The TLD, ancestor, and parent iterators supply their own evaluation filters and are left for later tiers, along with the proposed non-query-field reductions. New tests cover the filter behavior and the config round-trip.