CI for workflow-to-Python verification: pooled Python workers

[kz930]

Design of the verification itself is in #6072 — this is only about why it was slow and what fixed it.

How we check one operator

Three steps, and each one that needs Python used to start a brand-new Python process:

1. Run it the normal Texera way — the ground truth. Scala operators run in the JVM, no Python needed. Python-based operators (Sort, most charts) start a Python process.
2. Run the standalone Python script our translator generated for it — always Python.
3. Compare the two outputs — a small pandas script, also Python.

So one check = 2 Python starts for a Scala operator, 3 for a Python-based one. And we do one check per enum option (line chart mode = line / dots / line+dots), so a single operator often means several checks.

The problem

• Starting Python + importing pandas/pyamber: ~300 ms.
• The operator's actual work on our small test table: ~a few ms.
• So ~96% of the time was imports — repaid on every single run, across 148 discovered operators.

The fix

• Keep Python alive. Start it once, import once, then feed it jobs: one job in on stdin, result out on stdout. Every job after the first skips the imports.
• Batch. Many operators share one worker, so imports are paid per worker, not per run.
• One pool per role — Texera path, generated script, comparison. Three different worker scripts (and the first needs its own PYTHONPATH), so workers are never mixed between them.
• Run 4 at a time. Each pool keeps up to 4 workers; a worker takes one job at a time, since it changes directory per job.
• If a worker crashes, we drop it and run that one job the old way — never worse than before. A normal test failure isn't a crash; the worker reports it and stays alive. TEXERA_TEST_PYTHON_WORKER=0 turns pooling off entirely.
• In CI, it's a step in the existing platform-integration job for workflow-compiling-service (Python and the service are already there). The unit job skips it by tag and stays fast.

The result

Same suite, same 53 operators, same machine:

	test run	wall clock	CPU used
workers on	8.0 s	14.2 s	61 s
workers off	32.8 s	39.3 s	201 s

• ~4× faster, ~3× less CPU.
• CI is green — 53 operators at the last run.

How to test it

Point UDF_PYTHON_PATH at a Python 3.12 that has pandas, plotly, pyarrow and betterproto, then:

UDF_PYTHON_PATH="$PWD/../venv312/bin/python" sbt "WorkflowCompilingService/testOnly *OperatorBehaviorSpec"

[kz930]

@carloea2 @Yicong-Huang

[chenlica]

@carloea2 @Yicong-Huang : any input?

@kz930 Please lead the discussion and close it if when it's done.

[carloea2]

It looks good to me

[kz930]

If it looks good, then I would close this discussion. This is the current implementation so far.