test(workflow-operator): add unit test coverage for IntervalJoin, JoinUtils, and OperatorGroupConstants - #5845
Merged
Conversation
…nUtils, and OperatorGroupConstants Pin behavior of three join/utility classes: - IntervalJoinOpDesc: operatorInfo (Interval Join, JOIN_GROUP, two ordered inputs left/right with right depending on left); getPhysicalOp wires IntervalJoinOpExec + a HashPartition requirement on each join key; schema propagation merges left++right schemas, suffixing a conflicting attribute with #@1; field round-trip - JoinUtils.joinTuples: concatenates left+right fields; skips the named join-key attribute from the right; renames a right-side name conflict with #@1 - OperatorGroupConstants: pins the canonical group-name string values and the OperatorGroupOrderList hierarchy (relational subgroups under Data Cleaning; visualization subgroups under Visualization, in panel order) 12 new tests; scalafmtCheck + scalafixAll --check clean; no production-code changes.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds ScalaTest unit specs under common/workflow-operator/ to pin existing behavior of join-related descriptors/utilities and operator-group metadata, without changing production code.
Changes:
- Add
IntervalJoinOpDescSpecto pinIntervalJoinOpDescdefaults, operatorInfo wiring, schema propagation, and JSON round-trip behavior. - Add
JoinUtilsSpecto pinJoinUtils.joinTuplesconcatenation, skip-attribute, and conflict-renaming behavior. - Add
OperatorGroupConstantsSpecto pin operator group names and theOperatorGroupOrderListnesting/order.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/metadata/OperatorGroupConstantsSpec.scala | New spec for operator group constant values and group ordering/nesting |
| common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/intervalJoin/IntervalJoinOpDescSpec.scala | New spec covering IntervalJoin operator descriptor behavior and schema propagation |
| common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/hashJoin/JoinUtilsSpec.scala | New spec covering JoinUtils tuple-merge behavior (skip + rename conflicts) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5845 +/- ##
============================================
- Coverage 53.86% 53.86% -0.01%
+ Complexity 2756 2754 -2
============================================
Files 1099 1099
Lines 42541 42541
Branches 4577 4577
============================================
- Hits 22916 22913 -3
Misses 18290 18290
- Partials 1335 1338 +3
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 385 | 0.235 | 25,977/30,792/30,792 us | 🔴 +19.4% / 🟢 -12.0% |
| 🔴 | bs=100 sw=10 sl=64 | 791 | 0.483 | 125,429/152,258/152,258 us | 🔴 +10.6% / 🔴 +11.7% |
| ⚪ | bs=1000 sw=10 sl=64 | 921 | 0.562 | 1,090,982/1,123,263/1,123,263 us | ⚪ within ±5% / 🔴 +12.2% |
Baseline details
Latest main 6d31f46 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 385 tuples/sec | 410 tuples/sec | 410.82 tuples/sec | -6.1% | -6.3% |
| bs=10 sw=10 sl=64 | MB/s | 0.235 MB/s | 0.25 MB/s | 0.251 MB/s | -6.0% | -6.3% |
| bs=10 sw=10 sl=64 | p50 | 25,977 us | 21,753 us | 23,785 us | +19.4% | +9.2% |
| bs=10 sw=10 sl=64 | p95 | 30,792 us | 35,910 us | 34,980 us | -14.3% | -12.0% |
| bs=10 sw=10 sl=64 | p99 | 30,792 us | 35,910 us | 34,980 us | -14.3% | -12.0% |
| bs=100 sw=10 sl=64 | throughput | 791 tuples/sec | 830 tuples/sec | 891.94 tuples/sec | -4.7% | -11.3% |
| bs=100 sw=10 sl=64 | MB/s | 0.483 MB/s | 0.506 MB/s | 0.544 MB/s | -4.5% | -11.3% |
| bs=100 sw=10 sl=64 | p50 | 125,429 us | 119,669 us | 112,277 us | +4.8% | +11.7% |
| bs=100 sw=10 sl=64 | p95 | 152,258 us | 137,669 us | 139,802 us | +10.6% | +8.9% |
| bs=100 sw=10 sl=64 | p99 | 152,258 us | 137,669 us | 139,802 us | +10.6% | +8.9% |
| bs=1000 sw=10 sl=64 | throughput | 921 tuples/sec | 917 tuples/sec | 1,041 tuples/sec | +0.4% | -11.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.562 MB/s | 0.559 MB/s | 0.635 MB/s | +0.5% | -11.6% |
| bs=1000 sw=10 sl=64 | p50 | 1,090,982 us | 1,092,524 us | 972,714 us | -0.1% | +12.2% |
| bs=1000 sw=10 sl=64 | p95 | 1,123,263 us | 1,124,173 us | 1,023,057 us | -0.1% | +9.8% |
| bs=1000 sw=10 sl=64 | p99 | 1,123,263 us | 1,124,173 us | 1,023,057 us | -0.1% | +9.8% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,519.18,200,128000,385,0.235,25976.84,30791.50,30791.50
1,100,10,64,20,2526.97,2000,1280000,791,0.483,125428.91,152257.52,152257.52
2,1000,10,64,20,21722.70,20000,12800000,921,0.562,1090981.70,1123262.87,1123262.87…me value Address Copilot review feedback on apache#5845: the value-pinning test asserted only a subset of constants while its name implied full coverage. It now asserts all 28 group-name constants (Database/API/UDF/ML/Sklearn/HuggingFace + every visualization subgroup), matching the test name.
yangzhang75
pushed a commit
to yangzhang75/texera
that referenced
this pull request
Jun 24, 2026
…nUtils, and OperatorGroupConstants (apache#5845) ### What changes were proposed in this PR? Pin behavior of three join/utility classes in `common/workflow-operator/`. No production-code changes. | Spec | Source class | Tests | | --- | --- | --- | | `IntervalJoinOpDescSpec` | `IntervalJoinOpDesc` | 5 | | `JoinUtilsSpec` | `JoinUtils` (object) | 3 | | `OperatorGroupConstantsSpec` | `OperatorGroupConstants` (object) | 4 | **Behavior pinned — `IntervalJoinOpDesc`** | Surface | Contract | | --- | --- | | `operatorInfo` | `Interval Join`, JOIN_GROUP; two ordered inputs (`left table` at `PortIdentity()`, `right table` at `PortIdentity(1)` depending on the left); one output | | Field defaults | join-key attrs `null`; `constant == 10`; `includeLeftBound`/`includeRightBound == true` | | `getPhysicalOp` | wires `IntervalJoinOpExec`; port identities carried; `HashPartition` requirement on each join key | | Schema propagation | merges left ⧺ right schemas, suffixing a conflicting attribute with `#@1` | **Behavior pinned — `JoinUtils.joinTuples`** | Surface | Contract | | --- | --- | | Concatenation | left + right fields merged | | Skip | the named join-key attribute is dropped from the right side | | Conflict rename | a right-side name collision is renamed with a `#@1` suffix | **Behavior pinned — `OperatorGroupConstants`** | Surface | Contract | | --- | --- | | Constant values | the canonical group-name strings (`Data Input`, `Join`, `Set`, … `Control Block`) | | `OperatorGroupOrderList` | starts at `Data Input`, ends at `Control Block`; relational subgroups (Join/Set/Aggregate/Sort) nested under `Data Cleaning`; visualization subgroups nested under `Visualization` in panel order | ### Any related issues, documentation, discussions? Closes apache#5840. ### How was this PR tested? - `sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec org.apache.texera.amber.operator.hashJoin.JoinUtilsSpec org.apache.texera.amber.operator.metadata.OperatorGroupConstantsSpec"` — 12 tests, all green - `sbt "WorkflowOperator/Test/scalafmtCheck"` and `sbt "WorkflowOperator/Test/scalafix --check"` — clean - CI to confirm ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8 [1M context])
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Pin behavior of three join/utility classes in
common/workflow-operator/. No production-code changes.IntervalJoinOpDescSpecIntervalJoinOpDescJoinUtilsSpecJoinUtils(object)OperatorGroupConstantsSpecOperatorGroupConstants(object)Behavior pinned —
IntervalJoinOpDescoperatorInfoInterval Join, JOIN_GROUP; two ordered inputs (left tableatPortIdentity(),right tableatPortIdentity(1)depending on the left); one outputnull;constant == 10;includeLeftBound/includeRightBound == truegetPhysicalOpIntervalJoinOpExec; port identities carried;HashPartitionrequirement on each join key#@1Behavior pinned —
JoinUtils.joinTuples#@1suffixBehavior pinned —
OperatorGroupConstantsData Input,Join,Set, …Control Block)OperatorGroupOrderListData Input, ends atControl Block; relational subgroups (Join/Set/Aggregate/Sort) nested underData Cleaning; visualization subgroups nested underVisualizationin panel orderAny related issues, documentation, discussions?
Closes #5840.
How was this PR tested?
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec org.apache.texera.amber.operator.hashJoin.JoinUtilsSpec org.apache.texera.amber.operator.metadata.OperatorGroupConstantsSpec"— 12 tests, all greensbt "WorkflowOperator/Test/scalafmtCheck"andsbt "WorkflowOperator/Test/scalafix --check"— cleanWas this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8 [1M context])