Skip to content

test(workflow-operator): add spec for TextGenCodegen - #5789

Merged
aglinxinyuan merged 1 commit into
apache:mainfrom
EmilySun621:test/text-gen-codegen
Jun 19, 2026
Merged

test(workflow-operator): add spec for TextGenCodegen#5789
aglinxinyuan merged 1 commit into
apache:mainfrom
EmilySun621:test/text-gen-codegen

Conversation

@EmilySun621

@EmilySun621 EmilySun621 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Adds a unit test spec for TextGenCodegen. Tests cover:

  • task equals "text-generation"
  • payloadPython(ctx) opens with the if task == "text-generation": branch
  • payloadPython(ctx) emits the chat-completions shape (messages, max_tokens, temperature)
  • payloadPython(ctx) includes the else fallback with {"inputs": prompt_value}
  • parsePython(ctx) reads from choices / message / content
  • No raw-string leakage: snippets reference self.* attributes, never literal context values
  • Context-independence: two different CodegenContext inputs produce identical output

Any related issues, documentation, discussions?

Closes #5778

How was this PR tested?

Spec verified with sbt "WorkflowOperator/testOnly *TextGenCodegenSpec". 7 tests passing.

Was this PR authored or co-authored using generative AI tooling?

Coauthored-by: Claude Code (Anthropic)

@codecov-commenter

codecov-commenter commented Jun 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 53.28%. Comparing base (731d671) to head (fd7363b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5789      +/-   ##
============================================
- Coverage     53.30%   53.28%   -0.03%     
+ Complexity     2668     2667       -1     
============================================
  Files          1098     1098              
  Lines         42532    42532              
  Branches       4575     4575              
============================================
- Hits          22673    22664       -9     
- Misses        18530    18537       +7     
- Partials       1329     1331       +2     
Flag Coverage Δ *Carryforward flag
access-control-service 70.44% <ø> (ø)
agent-service 34.36% <ø> (ø) Carriedforward from 731d671
amber 53.76% <ø> (-0.06%) ⬇️
computing-unit-managing-service 1.65% <ø> (ø)
config-service 56.71% <ø> (ø)
file-service 57.06% <ø> (ø)
frontend 48.00% <ø> (ø) Carriedforward from 731d671
pyamber 90.13% <ø> (ø) Carriedforward from 731d671
python 90.80% <ø> (ø) Carriedforward from 731d671
workflow-compiling-service 58.69% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 4 worse · ⚪ 11 noise (<±5%) · 0 without baseline

Compared against main 731d671 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 432 0.263 22,329/32,550/32,550 us 🔴 +13.3% / 🟢 -6.9%
bs=100 sw=10 sl=64 922 0.563 104,126/143,971/143,971 us ⚪ within ±5% / 🟢 -7.3%
bs=1000 sw=10 sl=64 1,121 0.684 894,797/931,763/931,763 us ⚪ within ±5% / 🟢 -8.9%
Baseline details

Latest main 731d671 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 432 tuples/sec 477 tuples/sec 410.82 tuples/sec -9.4% +5.2%
bs=10 sw=10 sl=64 MB/s 0.263 MB/s 0.291 MB/s 0.251 MB/s -9.6% +4.9%
bs=10 sw=10 sl=64 p50 22,329 us 21,863 us 23,785 us +2.1% -6.1%
bs=10 sw=10 sl=64 p95 32,550 us 28,724 us 34,980 us +13.3% -6.9%
bs=10 sw=10 sl=64 p99 32,550 us 28,724 us 34,980 us +13.3% -6.9%
bs=100 sw=10 sl=64 throughput 922 tuples/sec 948 tuples/sec 891.94 tuples/sec -2.7% +3.4%
bs=100 sw=10 sl=64 MB/s 0.563 MB/s 0.579 MB/s 0.544 MB/s -2.8% +3.4%
bs=100 sw=10 sl=64 p50 104,126 us 104,160 us 112,277 us -0.0% -7.3%
bs=100 sw=10 sl=64 p95 143,971 us 150,671 us 139,802 us -4.4% +3.0%
bs=100 sw=10 sl=64 p99 143,971 us 150,671 us 139,802 us -4.4% +3.0%
bs=1000 sw=10 sl=64 throughput 1,121 tuples/sec 1,106 tuples/sec 1,041 tuples/sec +1.4% +7.7%
bs=1000 sw=10 sl=64 MB/s 0.684 MB/s 0.675 MB/s 0.635 MB/s +1.3% +7.6%
bs=1000 sw=10 sl=64 p50 894,797 us 902,236 us 972,714 us -0.8% -8.0%
bs=1000 sw=10 sl=64 p95 931,763 us 975,908 us 1,023,057 us -4.5% -8.9%
bs=1000 sw=10 sl=64 p99 931,763 us 975,908 us 1,023,057 us -4.5% -8.9%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,463.47,200,128000,432,0.263,22328.61,32550.29,32550.29
1,100,10,64,20,2168.16,2000,1280000,922,0.563,104125.65,143970.51,143970.51
2,1000,10,64,20,17848.62,20000,12800000,1121,0.684,894797.20,931762.75,931762.75

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds targeted ScalaTest coverage for the Hugging Face TextGenCodegen Python snippet generator (payload + response parsing) in the common/workflow-operator test suite, pinning the expected “text-generation” / chat-completions behavior and guarding against accidental context value inlining.

Changes:

  • Introduces TextGenCodegenSpec validating task, key payload fields/branches, and parse output structure.
  • Adds regression checks ensuring no CodegenContext string values are inlined and that outputs are context-independent.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@aglinxinyuan

Copy link
Copy Markdown
Contributor

For PR description, use Closes instead of related to link to the issue. related doesn't work.

@EmilySun621

Copy link
Copy Markdown
Contributor Author

For PR description, use Closes instead of related to link to the issue. related doesn't work.

Thanks for the review. I changed the "related" to "Closes".

@aglinxinyuan aglinxinyuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Jun 19, 2026
Merged via the queue into apache:main with commit ba32e2b Jun 19, 2026
38 of 40 checks passed
yangzhang75 pushed a commit to yangzhang75/texera that referenced this pull request Jun 24, 2026
<!--
Thanks for sending a pull request (PR)! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
[Contributing to
Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md)
  2. Ensure you have added or run the appropriate tests for your PR
  3. If the PR is work in progress, mark it a draft on GitHub.
  4. Please write your PR title to summarize what this PR proposes, we 
    are following Conventional Commits style for PR titles as well.
  5. Be sure to keep the PR description updated to reflect all changes.
-->

### What changes were proposed in this PR?
<!--
Please clarify what changes you are proposing. The purpose of this
section
is to outline the changes. Here are some tips for you:
  1. If you propose a new API, clarify the use case for a new API.
  2. If you fix a bug, you can clarify why it is a bug.
  3. If it is a refactoring, clarify what has been changed.
  3. It would be helpful to include a before-and-after comparison using 
     screenshots or GIFs.
  4. Please consider writing useful notes for better and faster reviews.
-->
Adds a unit test spec for `TextGenCodegen`. Tests cover:
- `task` equals `"text-generation"`
- `payloadPython(ctx)` opens with the `if task == "text-generation":`
branch
- `payloadPython(ctx)` emits the chat-completions shape (`messages`,
`max_tokens`, `temperature`)
- `payloadPython(ctx)` includes the `else` fallback with `{"inputs":
prompt_value}`
- `parsePython(ctx)` reads from `choices` / `message` / `content`
- No raw-string leakage: snippets reference `self.*` attributes, never
literal context values
- Context-independence: two different `CodegenContext` inputs produce
identical output

### Any related issues, documentation, discussions?
<!--
Please use this section to link other resources if not mentioned
already.
1. If this PR fixes an issue, please include `Fixes apache#1234`, `Resolves
apache#1234`
or `Closes apache#1234`. If it is only related, simply mention the issue
number.
  2. If there is design documentation, please add the link.
  3. If there is a discussion in the mailing list, please add the link.
-->
Closes apache#5778

### How was this PR tested?
<!--
If tests were added, say they were added here. Or simply mention that if
the PR
is tested with existing test cases. Make sure to include/update test
cases that
check the changes thoroughly including negative and positive cases if
possible.
If it was tested in a way different from regular unit tests, please
clarify how
you tested step by step, ideally copy and paste-able, so that other
reviewers can
test and check, and descendants can verify in the future. If tests were
not added,
please describe why they were not added and/or why it was difficult to
add.
-->
Spec verified with `sbt "WorkflowOperator/testOnly
*TextGenCodegenSpec"`. 7 tests passing.

### Was this PR authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
PR,
please include the phrase: 'Generated-by: ' followed by the name of the
tool
and its version. If no, write 'No'. 
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for
details.
-->
Coauthored-by: Claude Code (Anthropic)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add unit test coverage for TextGenCodegen

4 participants