test(workflow-operator): add unit test coverage for visualization operator descriptors by aglinxinyuan · Pull Request #5827 · apache/texera

aglinxinyuan · 2026-06-20T07:15:15Z

What changes were proposed in this PR?

Pin behavior of three previously-untested visualization descriptors in common/workflow-operator/. No production-code changes.

Spec	Source class	Tests
`HtmlVizOpDescSpec`	`HtmlVizOpDesc`	4
`CandlestickChartOpDescSpec`	`CandlestickChartOpDesc`	5
`Histogram2DOpDescSpec`	`Histogram2DOpDesc`	6

All three spec files follow the <srcClassName>Spec.scala one-to-one convention.

Behavior pinned

Surface	Contract
`operatorInfo`	exact name + visualization group (`MEDIA` / `FINANCIAL` / `STATISTICAL`); one input / one output
Output schema	all three emit a single `html-content` STRING column (`HtmlViz` via `getExternalOutputSchemas`; charts via `getOutputSchemas`)
`getPhysicalOp` wiring (`HtmlViz`)	`OpExecWithClassName("…htmlviz.HtmlVizOpExec")`; port identities carried forward
Field defaults	`Candlestick` OHLC columns default `""`; `Histogram2D` `xBins`/`yBins == 10`, `normalize == DENSITY`
`generatePythonCode`	`Candlestick` emits a Plotly `go.Candlestick(` figure; `Histogram2D` emits `px.density_heatmap(` and rejects a non-positive bin count (`AssertionError`)
Round-trip	all config fields preserved through the polymorphic base

The specs pin the stable contract (operatorInfo + output schema + codegen guards) rather than the full Plotly template, and never assert on interpolated EncodableString values (which are decoded at runtime, not embedded raw).

Any related issues, documentation, discussions?

Closes #5824.

How was this PR tested?

Pure unit-test additions; verified locally with:

sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.visualization.htmlviz.HtmlVizOpDescSpec org.apache.texera.amber.operator.visualization.candlestickChart.CandlestickChartOpDescSpec org.apache.texera.amber.operator.visualization.histogram2d.Histogram2DOpDescSpec" — 15 tests, all green
sbt "WorkflowOperator/Test/scalafmtCheck" and sbt "WorkflowOperator/Test/scalafix --check" — clean
CI to confirm

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8 [1M context])

…rator descriptors Pin behavior of three previously-untested visualization descriptors: - HtmlVizOpDesc: operatorInfo (HTML Visualizer, VISUALIZATION_MEDIA_GROUP); wires HtmlVizOpExec; schema propagation emits a single html-content STRING column; htmlContentAttrName default - CandlestickChartOpDesc: operatorInfo (Candlestick Chart, VISUALIZATION_FINANCIAL_GROUP); getOutputSchemas html-content STRING; OHLC field defaults; generatePythonCode emits a Plotly Candlestick figure; field round-trip - Histogram2DOpDesc: operatorInfo (Histogram2D, VISUALIZATION_STATISTICAL_GROUP); getOutputSchemas html-content STRING; defaults (xBins/yBins=10, normalize=DENSITY); generatePythonCode rejects a non-positive bin count; field round-trip getPhysicalOp tests (HtmlViz) assert port IDENTITIES (keySet) rather than counts. 15 new tests; scalafmtCheck + scalafixAll --check clean; no production-code changes.

codecov-commenter · 2026-06-20T07:17:56Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 53.58%. Comparing base (4722a42) to head (6ac46eb).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #5827      +/-   ##
============================================
+ Coverage     53.44%   53.58%   +0.14%     
- Complexity     2710     2727      +17     
============================================
  Files          1099     1099              
  Lines         42541    42541              
  Branches       4577     4577              
============================================
+ Hits          22735    22796      +61     
+ Misses        18474    18410      -64     
- Partials       1332     1335       +3

Flag	Coverage Δ		*Carryforward flag
access-control-service	`70.44% <ø> (ø)`
agent-service	`34.36% <ø> (ø)`		Carriedforward from 77e5b3a
amber	`54.47% <ø> (+0.36%)`	⬆️
computing-unit-managing-service	`1.65% <ø> (ø)`
config-service	`56.71% <ø> (ø)`
file-service	`57.06% <ø> (ø)`
frontend	`48.05% <ø> (ø)`		Carriedforward from 77e5b3a
pyamber	`90.13% <ø> (ø)`		Carriedforward from 77e5b3a
python	`90.80% <ø> (ø)`		Carriedforward from 77e5b3a
workflow-compiling-service	`58.69% <ø> (ø)`

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

Adds ScalaTest unit coverage for three previously untested visualization operator descriptors under common/workflow-operator/, pinning their public contracts (operator metadata, output schema, codegen guards, and JSON round-trip) without modifying production code.

Changes:

Introduce HtmlVizOpDescSpec covering operatorInfo, physical wiring, and schema propagation.
Introduce CandlestickChartOpDescSpec covering defaults, output schema, codegen smoke assertions, and JSON polymorphic round-trip.
Introduce Histogram2DOpDescSpec covering defaults, output schema, codegen guard for invalid bins, and JSON polymorphic round-trip.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/htmlviz/HtmlVizOpDescSpec.scala	Adds unit tests for HtmlViz descriptor metadata, physical-op wiring, and schema propagation.
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/candlestickChart/CandlestickChartOpDescSpec.scala	Adds unit tests for Candlestick descriptor metadata, defaults, output schema, codegen smoke check, and JSON round-trip.
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/histogram2d/Histogram2DOpDescSpec.scala	Adds unit tests for Histogram2D descriptor metadata, defaults, output schema, codegen guard, and JSON round-trip.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-06-20T07:20:42Z

✅ No material benchmark regressions detected

🟢 4 better · 🔴 0 worse · ⚪ 11 noise (<±5%) · 0 without baseline

Compared against main 41123c0 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🟢	bs=10 sw=10 sl=64	431	0.263	22,732/31,421/31,421 us	🟢 -6.5% / 🟢 -10.2%
🟢	bs=100 sw=10 sl=64	969	0.591	103,395/114,113/114,113 us	🟢 -18.3% / 🟢 -18.4%
⚪	bs=1000 sw=10 sl=64	1,115	0.68	900,116/951,951/951,951 us	⚪ within ±5% / 🟢 -7.5%

Baseline details

Latest main 41123c0 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	431 tuples/sec	440 tuples/sec	410.82 tuples/sec	-2.0%	+4.9%
bs=10 sw=10 sl=64	MB/s	0.263 MB/s	0.269 MB/s	0.251 MB/s	-2.2%	+4.9%
bs=10 sw=10 sl=64	p50	22,732 us	21,877 us	23,785 us	+3.9%	-4.4%
bs=10 sw=10 sl=64	p95	31,421 us	33,609 us	34,980 us	-6.5%	-10.2%
bs=10 sw=10 sl=64	p99	31,421 us	33,609 us	34,980 us	-6.5%	-10.2%
bs=100 sw=10 sl=64	throughput	969 tuples/sec	937 tuples/sec	891.94 tuples/sec	+3.4%	+8.6%
bs=100 sw=10 sl=64	MB/s	0.591 MB/s	0.572 MB/s	0.544 MB/s	+3.3%	+8.6%
bs=100 sw=10 sl=64	p50	103,395 us	105,880 us	112,277 us	-2.3%	-7.9%
bs=100 sw=10 sl=64	p95	114,113 us	139,670 us	139,802 us	-18.3%	-18.4%
bs=100 sw=10 sl=64	p99	114,113 us	139,670 us	139,802 us	-18.3%	-18.4%
bs=1000 sw=10 sl=64	throughput	1,115 tuples/sec	1,106 tuples/sec	1,041 tuples/sec	+0.8%	+7.1%
bs=1000 sw=10 sl=64	MB/s	0.68 MB/s	0.675 MB/s	0.635 MB/s	+0.7%	+7.0%
bs=1000 sw=10 sl=64	p50	900,116 us	901,864 us	972,714 us	-0.2%	-7.5%
bs=1000 sw=10 sl=64	p95	951,951 us	942,961 us	1,023,057 us	+1.0%	-7.0%
bs=1000 sw=10 sl=64	p99	951,951 us	942,961 us	1,023,057 us	+1.0%	-7.0%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,463.55,200,128000,431,0.263,22731.79,31420.97,31420.97
1,100,10,64,20,2064.30,2000,1280000,969,0.591,103394.75,114113.47,114113.47
2,1000,10,64,20,17938.37,20000,12800000,1115,0.680,900116.23,951951.15,951951.15

…d output port in viz specs Address Copilot review feedback on apache#5827: the output-schema tests asserted only .values.head, so a regression that keyed the schema under the wrong PortIdentity would still pass as long as there was a single entry. Assert the full Map is keyed by operatorInfo.outputPorts.head.id (and feed the input under operatorInfo.inputPorts.head.id) in HtmlViz / Candlestick / Histogram2D specs.

…d output port in column-transform specs Proactively apply the same strengthening Copilot requested on the visualization specs (apache#5827): TypeCasting and UnnestString schema-propagation tests asserted only .values.head, which passes even if the schema is produced under the wrong PortIdentity. Assert the full Map keyed by operatorInfo.outputPorts.head.id and feed the input under operatorInfo.inputPorts.head.id.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

@JsonProperty

…tions + round-trip Address Copilot review feedback on apache#5827: the issue/PR scope said HtmlVizOpDescSpec pins htmlContentAttrName's annotations and that the config round-trips, but the spec only checked the default value. Add reflection-based assertions for @JsonProperty(required = true) + @AutofillAttributeName + @NotNull('HTML content cannot be empty') (mirroring UrlVizOpDescSpec) and a polymorphic JSON round-trip of htmlContentAttrName, so the spec matches the stated scope.

xuang7

LGTM!

…rator descriptors (apache#5827) ### What changes were proposed in this PR? Pin behavior of three previously-untested visualization descriptors in `common/workflow-operator/`. No production-code changes. | Spec | Source class | Tests | | --- | --- | --- | | `HtmlVizOpDescSpec` | `HtmlVizOpDesc` | 4 | | `CandlestickChartOpDescSpec` | `CandlestickChartOpDesc` | 5 | | `Histogram2DOpDescSpec` | `Histogram2DOpDesc` | 6 | All three spec files follow the `<srcClassName>Spec.scala` one-to-one convention. **Behavior pinned** | Surface | Contract | | --- | --- | | `operatorInfo` | exact name + visualization group (`MEDIA` / `FINANCIAL` / `STATISTICAL`); one input / one output | | Output schema | all three emit a single `html-content` STRING column (`HtmlViz` via `getExternalOutputSchemas`; charts via `getOutputSchemas`) | | `getPhysicalOp` wiring (`HtmlViz`) | `OpExecWithClassName("…htmlviz.HtmlVizOpExec")`; port **identities** carried forward | | Field defaults | `Candlestick` OHLC columns default `""`; `Histogram2D` `xBins`/`yBins == 10`, `normalize == DENSITY` | | `generatePythonCode` | `Candlestick` emits a Plotly `go.Candlestick(` figure; `Histogram2D` emits `px.density_heatmap(` and **rejects a non-positive bin count** (`AssertionError`) | | Round-trip | all config fields preserved through the polymorphic base | The specs pin the stable contract (operatorInfo + output schema + codegen guards) rather than the full Plotly template, and never assert on interpolated `EncodableString` values (which are decoded at runtime, not embedded raw). ### Any related issues, documentation, discussions? Closes apache#5824. ### How was this PR tested? Pure unit-test additions; verified locally with: - `sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.visualization.htmlviz.HtmlVizOpDescSpec org.apache.texera.amber.operator.visualization.candlestickChart.CandlestickChartOpDescSpec org.apache.texera.amber.operator.visualization.histogram2d.Histogram2DOpDescSpec"` — 15 tests, all green - `sbt "WorkflowOperator/Test/scalafmtCheck"` and `sbt "WorkflowOperator/Test/scalafix --check"` — clean - CI to confirm ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8 [1M context])

Copilot AI review requested due to automatic review settings June 20, 2026 07:15

github-actions Bot assigned aglinxinyuan Jun 20, 2026

github-actions Bot added the common label Jun 20, 2026

Copilot started reviewing on behalf of aglinxinyuan June 20, 2026 07:15 View session

Copilot AI reviewed Jun 20, 2026

View reviewed changes

aglinxinyuan requested review from Copilot and mengw15 June 20, 2026 21:35

Copilot started reviewing on behalf of aglinxinyuan June 20, 2026 21:35 View session

Copilot AI reviewed Jun 20, 2026

View reviewed changes

Comment thread ...rc/test/scala/org/apache/texera/amber/operator/visualization/htmlviz/HtmlVizOpDescSpec.scala

aglinxinyuan requested a review from xuang7 June 20, 2026 23:15

xuang7 approved these changes Jun 21, 2026

View reviewed changes

aglinxinyuan added this pull request to the merge queue Jun 21, 2026

Merged via the queue into apache:main with commit 6d31f46 Jun 21, 2026
20 checks passed

aglinxinyuan deleted the test-visualization-descriptors branch June 21, 2026 04:13

Navigation Menu

Uh oh!

test(workflow-operator): add unit test coverage for visualization operator descriptors - #5827

test(workflow-operator): add unit test coverage for visualization operator descriptors#5827
aglinxinyuan merged 3 commits into
apache:mainfrom
aglinxinyuan:test-visualization-descriptors

aglinxinyuan commented Jun 20, 2026

Uh oh!

codecov-commenter commented Jun 20, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

xuang7 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

aglinxinyuan commented Jun 20, 2026

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

codecov-commenter commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ No material benchmark regressions detected

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

xuang7 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading