Skip to content

Do not merge patterns with exclusions#149387

Open
idegtiarenko wants to merge 3 commits into
elastic:mainfrom
idegtiarenko:do_not_merge_patterns_with_exclusions
Open

Do not merge patterns with exclusions#149387
idegtiarenko wants to merge 3 commits into
elastic:mainfrom
idegtiarenko:do_not_merge_patterns_with_exclusions

Conversation

@idegtiarenko
Copy link
Copy Markdown
Contributor

This change prevents merging unions into a single pattern if any of them contain exclusion.

For example with the following setup

        addIndex("data-ll");
        addIndex("data-rr");
        addView("data-l", "FROM index-l");
        addView("data-r", "FROM index-r");

A query FROM -*l,data-*
was resolved as FROM index-l,index-r,-*l,data-*
where index-l was excluded by -*l.

Now it is correctly resolved as FROM (FROM index-l,index-r),(FROM -*l,data-*) containing the scope of -*l exclusion.

@idegtiarenko idegtiarenko added >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL v9.5.0 labels May 19, 2026
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

// can not merge if any contains exclusion. This risk expanding scope of exclusion to unrelated pattern
if (patternsContainsExclusion(mainPatterns) || patternsContainsExclusion(otherPatterns)) {
return null;
}
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could still merge patterns if exclusion is only present in the first pattern, but not in the second one. Not sure if it could cause other issues? Maybe with deterministic execution?

@n1v0lg maybe you have an opinion on this?

Copy link
Copy Markdown
Contributor

@craigtaverner craigtaverner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - Although I think we can also consider stripping leading exclusions in general, as a separate refinement (but that will require changing this test).

addIndex("logs3");
LogicalPlan plan = query("FROM logs*,-logs3");
assertThat(replaceViews(plan), matchesPlan(query("FROM logs2,logs*,-logs3")));
assertThat(replaceViews(plan), matchesPlan(query("FROM (FROM logs2),(FROM logs*,-logs3)")));
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This highlights one change in behavior.

FROM logs* matches both logs2 index and logs1 view that effectively points to logs2 index.

With exclusion the patterns are kept separate in 2 subqueries. This would effectively read from logs2 twice.

If we remove the exclusion, the pattern would be coalesced to logs2,logs* since we no longer need to scope exclusion. As a result logs2 would be resolved and queried only once.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is intended to query index twice if it is referenced directly by a pattern and by a view, but I do not think that we can do that reliably with patterns.
Maybe that could be achieved if we never merge relations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants