Do not merge patterns with exclusions#149387
Conversation
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
| // can not merge if any contains exclusion. This risk expanding scope of exclusion to unrelated pattern | ||
| if (patternsContainsExclusion(mainPatterns) || patternsContainsExclusion(otherPatterns)) { | ||
| return null; | ||
| } |
There was a problem hiding this comment.
I think we could still merge patterns if exclusion is only present in the first pattern, but not in the second one. Not sure if it could cause other issues? Maybe with deterministic execution?
@n1v0lg maybe you have an opinion on this?
craigtaverner
left a comment
There was a problem hiding this comment.
LGTM - Although I think we can also consider stripping leading exclusions in general, as a separate refinement (but that will require changing this test).
| addIndex("logs3"); | ||
| LogicalPlan plan = query("FROM logs*,-logs3"); | ||
| assertThat(replaceViews(plan), matchesPlan(query("FROM logs2,logs*,-logs3"))); | ||
| assertThat(replaceViews(plan), matchesPlan(query("FROM (FROM logs2),(FROM logs*,-logs3)"))); |
There was a problem hiding this comment.
This highlights one change in behavior.
FROM logs* matches both logs2 index and logs1 view that effectively points to logs2 index.
With exclusion the patterns are kept separate in 2 subqueries. This would effectively read from logs2 twice.
If we remove the exclusion, the pattern would be coalesced to logs2,logs* since we no longer need to scope exclusion. As a result logs2 would be resolved and queried only once.
There was a problem hiding this comment.
I think it is intended to query index twice if it is referenced directly by a pattern and by a view, but I do not think that we can do that reliably with patterns.
Maybe that could be achieved if we never merge relations.
This change prevents merging unions into a single pattern if any of them contain exclusion.
For example with the following setup
A query
FROM -*l,data-*was resolved as
FROM index-l,index-r,-*l,data-*where
index-lwas excluded by-*l.Now it is correctly resolved as
FROM (FROM index-l,index-r),(FROM -*l,data-*)containing the scope of-*lexclusion.