test(workflow-operator): add unit test coverage for operator metadata types (OperatorInfo, PortDescription, PortDescriptor) - #5832
Conversation
… types (OperatorInfo, PortDescription, PortDescriptor) Pin behavior of three core operator-metadata types: - OperatorInfo (case class): constructor field exposure; the four boolean flags (dynamicInputPorts/dynamicOutputPorts/supportReconfiguration/allowPortCustomization) default to false and round-trip true; OperatorInfo.forVisualization builds a single disallow-multi-links input and a single SINGLE_SNAPSHOT output; value equality - PortDescription (case class): field exposure; dependencies default List.empty + accepts explicit deps; value equality + copy; @JsonIgnoreProperties('allowMultiInputs') backward-compat marker - PortDescriptor (trait): inputPorts/outputPorts default to null (not empty) and are reassignable Scoped to the pure case-class/trait types only; the impure OperatorMetadataGenerator object (eager reflective registry init) is deliberately not touched. 11 new tests; scalafmtCheck + scalafixAll --check clean; no production-code changes.
There was a problem hiding this comment.
Pull request overview
Adds Scala unit tests in common/workflow-operator to pin the current behavior of three core operator-metadata types (OperatorInfo, PortDescription, PortDescriptor) that are widely used for operator port wiring and metadata across LogicalOp implementations. No production code changes are included.
Changes:
- Add
OperatorInfoSpecto validate constructor field exposure, boolean default/override behavior,forVisualizationfactory contract, and case-class equality. - Add
PortDescriptionSpecto validate field exposure,dependenciesdefault/override, copy/equality behavior, and the@JsonIgnoreProperties("allowMultiInputs")backward-compat marker. - Add
PortDescriptorSpecto validate the trait’sinputPorts/outputPortsdefault-to-nullbehavior and reassignment semantics.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/PortDescriptorSpec.scala | Adds unit tests pinning PortDescriptor default null ports and reassignment behavior. |
| common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/PortDescriptionSpec.scala | Adds unit tests for PortDescription defaults, equality/copy, and Jackson backward-compat annotation presence. |
| common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/metadata/OperatorInfoSpec.scala | Adds unit tests for OperatorInfo field exposure/defaults, forVisualization contract, and value-based equality. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5832 +/- ##
============================================
+ Coverage 53.56% 53.62% +0.05%
- Complexity 2717 2720 +3
============================================
Files 1099 1099
Lines 42541 42541
Branches 4577 4577
============================================
+ Hits 22789 22814 +25
+ Misses 18421 18392 -29
- Partials 1331 1335 +4
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 363 | 0.222 | 27,683/34,434/34,434 us | 🔴 +37.6% / 🔴 +16.4% |
| 🔴 | bs=100 sw=10 sl=64 | 933 | 0.57 | 107,937/144,823/144,823 us | 🔴 +6.3% / ⚪ within ±5% |
| 🟢 | bs=1000 sw=10 sl=64 | 1,115 | 0.681 | 894,317/946,639/946,639 us | 🟢 -9.1% / 🟢 -8.1% |
Baseline details
Latest main 357fed0 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 363 tuples/sec | 456 tuples/sec | 410.82 tuples/sec | -20.4% | -11.6% |
| bs=10 sw=10 sl=64 | MB/s | 0.222 MB/s | 0.278 MB/s | 0.251 MB/s | -20.1% | -11.5% |
| bs=10 sw=10 sl=64 | p50 | 27,683 us | 20,118 us | 23,785 us | +37.6% | +16.4% |
| bs=10 sw=10 sl=64 | p95 | 34,434 us | 34,624 us | 34,980 us | -0.5% | -1.6% |
| bs=10 sw=10 sl=64 | p99 | 34,434 us | 34,624 us | 34,980 us | -0.5% | -1.6% |
| bs=100 sw=10 sl=64 | throughput | 933 tuples/sec | 939 tuples/sec | 891.94 tuples/sec | -0.6% | +4.6% |
| bs=100 sw=10 sl=64 | MB/s | 0.57 MB/s | 0.573 MB/s | 0.544 MB/s | -0.5% | +4.7% |
| bs=100 sw=10 sl=64 | p50 | 107,937 us | 102,190 us | 112,277 us | +5.6% | -3.9% |
| bs=100 sw=10 sl=64 | p95 | 144,823 us | 136,294 us | 139,802 us | +6.3% | +3.6% |
| bs=100 sw=10 sl=64 | p99 | 144,823 us | 136,294 us | 139,802 us | +6.3% | +3.6% |
| bs=1000 sw=10 sl=64 | throughput | 1,115 tuples/sec | 1,085 tuples/sec | 1,041 tuples/sec | +2.8% | +7.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.681 MB/s | 0.662 MB/s | 0.635 MB/s | +2.9% | +7.2% |
| bs=1000 sw=10 sl=64 | p50 | 894,317 us | 911,100 us | 972,714 us | -1.8% | -8.1% |
| bs=1000 sw=10 sl=64 | p95 | 946,639 us | 1,041,648 us | 1,023,057 us | -9.1% | -7.5% |
| bs=1000 sw=10 sl=64 | p99 | 946,639 us | 1,041,648 us | 1,023,057 us | -9.1% | -7.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,550.43,200,128000,363,0.222,27683.15,34434.30,34434.30
1,100,10,64,20,2142.97,2000,1280000,933,0.570,107936.70,144823.17,144823.17
2,1000,10,64,20,17937.02,20000,12800000,1115,0.681,894317.09,946638.93,946638.93…ants (apache#5739) ### What changes were proposed in this PR? Adds a unit-test spec for `PropertyNameConstants` (the `@JsonProperty` wire-key constants shared across `LogicalOp` serialization). No production-code changes. | Spec | Source class | Tests | | --- | --- | --- | | `PropertyNameConstantsSpec` | `PropertyNameConstants` | 5 | > **Scope note:** this PR originally also bundled `OperatorDescriptorUtils` and `PortDescriptor`/`PortDescription` specs. Both have since been covered on `main` / by other open PRs: > - `OperatorDescriptorUtilsSpec` already merged via apache#5798 — dropped here (it was the merge conflict). > - `PortDescriptor` + `PortDescription` are covered by apache#5832 with one spec file per source class (the repo convention) — dropped here to avoid a duplicate `PortDescriptorSpec.scala`. > > This PR is now scoped to its unique contribution, `PropertyNameConstants`, and is rebased on current `main`. ### Any related issues, documentation, discussions? Follow-up test coverage; see apache#5798 and apache#5832 for the de-duplicated specs. ### How was this PR tested? - `sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.metadata.PropertyNameConstantsSpec"` — 5 tests, all green - `sbt "WorkflowOperator/Test/scalafmtCheck"` and `sbt "WorkflowOperator/Test/scalafix --check"` — clean - CI to confirm ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8 [1M context])
… types (OperatorInfo, PortDescription, PortDescriptor) (apache#5832) ### What changes were proposed in this PR? Pin behavior of three core operator-metadata types in `common/workflow-operator/` — the data/trait types every `LogicalOp` relies on for port wiring and metadata. No production-code changes. | Spec | Source class | Tests | | --- | --- | --- | | `OperatorInfoSpec` | `OperatorInfo` (case class) | 4 | | `PortDescriptionSpec` | `PortDescription` (case class) | 5 | | `PortDescriptorSpec` | `PortDescriptor` (trait) | 2 | **Behavior pinned** | Surface | Contract | | --- | --- | | `OperatorInfo` fields | constructor field exposure; the four boolean flags default to `false` and round-trip `true` when set; value equality | | `OperatorInfo.forVisualization` | `inputPorts == List(InputPort(disallowMultiLinks = true))`, `outputPorts == List(OutputPort(mode = SINGLE_SNAPSHOT))` | | `PortDescription` | field exposure; `dependencies` defaults `List.empty` + accepts explicit deps; value equality + `copy`; `@JsonIgnoreProperties("allowMultiInputs")` backward-compat marker (verified via reflection) | | `PortDescriptor` (trait) | `inputPorts`/`outputPorts` default to `null` (not empty) and are reassignable | **Note for reviewers:** scoped to the pure case-class/trait types only. The `OperatorMetadataGenerator` object in the same file is deliberately **not** tested — it eagerly reflects over the entire `LogicalOp` registry at init (instantiating every descriptor), which is not pure-unit-testable. ### Any related issues, documentation, discussions? Closes apache#5829. ### How was this PR tested? - `sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.metadata.OperatorInfoSpec org.apache.texera.amber.operator.PortDescriptionSpec org.apache.texera.amber.operator.PortDescriptorSpec"` — 11 tests, all green - `sbt "WorkflowOperator/Test/scalafmtCheck"` and `sbt "WorkflowOperator/Test/scalafix --check"` — clean - CI to confirm ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8 [1M context])
…ants (apache#5739) ### What changes were proposed in this PR? Adds a unit-test spec for `PropertyNameConstants` (the `@JsonProperty` wire-key constants shared across `LogicalOp` serialization). No production-code changes. | Spec | Source class | Tests | | --- | --- | --- | | `PropertyNameConstantsSpec` | `PropertyNameConstants` | 5 | > **Scope note:** this PR originally also bundled `OperatorDescriptorUtils` and `PortDescriptor`/`PortDescription` specs. Both have since been covered on `main` / by other open PRs: > - `OperatorDescriptorUtilsSpec` already merged via apache#5798 — dropped here (it was the merge conflict). > - `PortDescriptor` + `PortDescription` are covered by apache#5832 with one spec file per source class (the repo convention) — dropped here to avoid a duplicate `PortDescriptorSpec.scala`. > > This PR is now scoped to its unique contribution, `PropertyNameConstants`, and is rebased on current `main`. ### Any related issues, documentation, discussions? Follow-up test coverage; see apache#5798 and apache#5832 for the de-duplicated specs. ### How was this PR tested? - `sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.metadata.PropertyNameConstantsSpec"` — 5 tests, all green - `sbt "WorkflowOperator/Test/scalafmtCheck"` and `sbt "WorkflowOperator/Test/scalafix --check"` — clean - CI to confirm ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8 [1M context])
What changes were proposed in this PR?
Pin behavior of three core operator-metadata types in
common/workflow-operator/— the data/trait types everyLogicalOprelies on for port wiring and metadata. No production-code changes.OperatorInfoSpecOperatorInfo(case class)PortDescriptionSpecPortDescription(case class)PortDescriptorSpecPortDescriptor(trait)Behavior pinned
OperatorInfofieldsfalseand round-triptruewhen set; value equalityOperatorInfo.forVisualizationinputPorts == List(InputPort(disallowMultiLinks = true)),outputPorts == List(OutputPort(mode = SINGLE_SNAPSHOT))PortDescriptiondependenciesdefaultsList.empty+ accepts explicit deps; value equality +copy;@JsonIgnoreProperties("allowMultiInputs")backward-compat marker (verified via reflection)PortDescriptor(trait)inputPorts/outputPortsdefault tonull(not empty) and are reassignableNote for reviewers: scoped to the pure case-class/trait types only. The
OperatorMetadataGeneratorobject in the same file is deliberately not tested — it eagerly reflects over the entireLogicalOpregistry at init (instantiating every descriptor), which is not pure-unit-testable.Any related issues, documentation, discussions?
Closes #5829.
How was this PR tested?
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.metadata.OperatorInfoSpec org.apache.texera.amber.operator.PortDescriptionSpec org.apache.texera.amber.operator.PortDescriptorSpec"— 11 tests, all greensbt "WorkflowOperator/Test/scalafmtCheck"andsbt "WorkflowOperator/Test/scalafix --check"— cleanWas this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8 [1M context])