Repository navigation
Fix CountStrategy returning capped counts when the count is observed - #3717
upadhyay-prashant wants to merge 1 commit into
Conversation
fa77f16 to
1b32af4
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3717 +/- ##
============================================
+ Coverage 76.35% 76.77% +0.41%
- Complexity 13424 14365 +941
============================================
Files 1012 1037 +25
Lines 60341 64834 +4493
Branches 7075 7708 +633
============================================
+ Hits 46076 49774 +3698
- Misses 11548 11970 +422
- Partials 2717 3090 +373 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1b32af4 to
eb17ead
Compare
eb17ead to
08d840e
Compare
CountStrategy rewrote count().is(P) to limit(n).count().is(P). When P accepts n, the capped value was emitted in place of the real count, so a following is(), the root traversal or a value-consuming parent saw the wrong number, e.g. where(out().count().is(gt(1)).is(lt(3))) kept vertices with more than two edges and out().count().is(gt(2)) returned 3. The limit is now only added when P rejects the cap, or when the count is used purely as a filter (where/filter, not, and/or, sideEffect, repeat until/emit) and no step follows is(). A label at the end of a where() child adds a WhereEndStep that compares the count to the path, so that case is not limited; other labels in a filter child can't be observed. The strategy also no longer touches count().is(P) when P contains a traversal, e.g. is(gt(1).and(lt(__.constant(3)))), since that bound is only resolved at runtime and no safe limit can be derived. Each case whose expectation changed has a counterpart that is still limited, so the productive cases stay covered.
08d840e to
2865a74
Compare
| {__.where(__.out().count().is(gt(1)).is(lt(3))), __.where(__.out().count().is(gt(1)).is(lt(3)))}, | ||
| {__.where(__.out().count().is(gte(2)).is(lte(2))), __.where(__.out().count().is(gte(2)).is(lte(2)))}, | ||
| {__.filter(__.out().count().is(neq(1)).is(lt(5))), __.filter(__.out().count().is(neq(1)).is(lt(5)))}, |
There was a problem hiding this comment.
I think theoretically these cases could be optimized as
| {__.where(__.out().count().is(gt(1)).is(lt(3))), __.where(__.out().count().is(gt(1)).is(lt(3)))}, | |
| {__.where(__.out().count().is(gte(2)).is(lte(2))), __.where(__.out().count().is(gte(2)).is(lte(2)))}, | |
| {__.filter(__.out().count().is(neq(1)).is(lt(5))), __.filter(__.out().count().is(neq(1)).is(lt(5)))}, | |
| {__.where(__.out().count().is(gt(1)).is(lt(3))), __.where(__.out().limit(3).count().is(gt(1)).is(lt(3)))}, | |
| {__.where(__.out().count().is(gte(2)).is(lte(2))), __.where(__.out().limit(3).count().is(gte(2)).is(lte(2)))}, | |
| {__.filter(__.out().count().is(neq(1)).is(lt(5))), __.filter(__.out().limit(5).count().is(neq(1)).is(lt(5)))}, |
I think the strategy is only looking at the count().is(gt/gte/neq).<some other step> pattern, which misses the context that the full chain of is() steps are all collectively constraining the count() in a way which could be optimized (the subsequent lt filter drops anything which would've exceeded the limit()). It's probably ok if we limit the scope of the strategy to the singular count().is() pattern, although I don't think these cases are necessarily warranted as they could be optimized further.
|
Thanks for all the updated @upadhyay-prashant. It looks great. I've just got the one comment where I think we could optimize further by inspecting chains of VOTE +1 |
| * Fixed `gremlin-go` to report a malformed or truncated GraphBinary response as a deserialization error rather than a bare decoder message. | ||
| * Made `TinkerGraph` an interface and renamed the in-memory implementation to `TinkerMemoryGraph`; `TinkerGraph.open()` and `gremlin.graph=...TinkerGraph` behave as before. *(breaking)* | ||
| * Renamed `TinkerTransactionGraph` to `TinkerStorageGraph`. *(breaking)* | ||
| * Fixed `CountStrategy` so that it no longer adds a `limit()` before `count().is(P)` when the capped count can pass the predicate and is then observed downstream, e.g. `where(out().count().is(gt(1)).is(lt(3)))` or `out().count().is(gt(2))`, or when `P` contains a traversal, e.g. `is(gt(1).and(lt(__.constant(3))))`. |
There was a problem hiding this comment.
Nit: Shorter changelog entries are preferred, perhaps:
| * Fixed `CountStrategy` so that it no longer adds a `limit()` before `count().is(P)` when the capped count can pass the predicate and is then observed downstream, e.g. `where(out().count().is(gt(1)).is(lt(3)))` or `out().count().is(gt(2))`, or when `P` contains a traversal, e.g. `is(gt(1).and(lt(__.constant(3))))`. | |
| * Fixed `CountStrategy` so that it no longer adds a `limit()` before `count().is(P)` when the capped count can pass the predicate and is then observed downstream, e.g. `out().count().is(gt(2))`. |
Summary
CountStrategyrewritescount().is(P)tolimit(n).count().is(P). WhenPacceptsn, the capped value is emitted in place of the real count, so anything that observes it gets a wrong result:g.V().where(outE().count().is(gt(1)).is(lt(3)))keeps vertices with more than 2 edgesg.V().outE().count().is(gt(2))returns3instead of6g.V().map(outE().count().is(gt(1)))returns2instead of3The limit is now only added when:
Prejects the cap (e.g.lt,lte,within), so any count that passes is exact; orwhere/filter,not,and/or,sideEffect,repeat().until()/emit()) and no step followsis(). A label at the end of awhere()child adds aWhereEndStepthat compares the count to the path, so that case is not limited.The strategy also leaves
count().is(P)alone whenPcontains a traversal (e.g.is(gt(1).and(lt(__.constant(3))))). That bound is only resolved at runtime, so no safe limit can be derived. This was already broken before this PR:filter(outE().count().is(gt(1).and(lt(__.constant(3)))))returned marko as well as josh, andoutE().count().as("a").is(gt(1).and(lte(__.select("a"))))returned2instead of6. A narrower rule (keep the limit when anand()has a literal part that rejects the cap) is possible as a follow-up.Otherwise the traversal is left unchanged. The existing
not()/ filter-removal rewrites are untouched.Testing
CountStrategyTest: added cases covering shapes that must not be limited and shapes that must stay optimized; every case whose expectation changed has a still-limited counterpart (e.g.out().count().is(neq(4))vswhere(out().count().is(neq(4)))). Also covers traversal-valued predicates in root, filter-only, label-reading andnot()forms.LazyBarrierStrategyTest/TraversalExplanationTest: updated two expectations that relied on the old root-level /by()rewrite;LazyBarrierStrategyTestalso gained awhere()counterpart.CountStrategy.feature: added 6 scenarios, all of which produced wrong results before the fix (except thelt(3).is(gt(1))control) and pass after.gremlin-coresuite andtinkergraph-gremlin(including its feature suites): 0 failures.translations.jsonand the Go/Python/.NET/JS translation files (additions only). GLV feature tests were not run locally.Notes
count().is(gt(n))no longer get the earlylimit(). That is required for correctness.3.7-dev/3.8-dev; this PR targetsmaster.