[pyoutline] Add OpenJD backend for job template serialization by mwiebe · Pull Request #2180 · AcademySoftwareFoundation/OpenCue

mwiebe · 2026-02-20T23:40:26Z

Link the Issue(s) this Pull Request is related to.

OpenJobDescription/openjd-model-for-python#137

Summarize your change.
Add a new PyOutline backend that serializes Outline jobs into OpenJD 2023-09 job templates. This enables running OpenCue jobs via the openjd CLI or submitting them to OpenJD-compatible schedulers like AWS Deadline Cloud. It doesn't look at the opposite of running an OpenJD job template in OpenCue. That's a separate follow-up to perform.

This PR doesn't fix the linked issue, but implementing OpenCue's pyoutline API for defining a job is a nice way to get a feel for how such an API might look in the openjd-model library, and evaluate any missing functionality in Open Job Description for supporting OpenCue jobs.

This PR depends on an in-progress RFC for the EXPR extension in Open Job Description, so it probably should stay in draft until that is merged there.

This PR was created with LLM assistance via Kiro CLI and the Claude Opus 4.6 model.

The changes include:

Serialize layers to OpenJD steps with CHUNK[INT] frame parameters
Map OpenCue command tokens (#IFRAME#, #FRAMESPEC#, etc.) to OpenJD expressions using the EXPR extension
Map host requirements: cores, memory, GPUs, tags, service, limits
Map dependencies including Post layer implicit depends
Wrap commands in bash scripts that export CUE_* environment variables
Support Shell, ShellScript, and PyEval layer types
Validate generated templates using openjd-model
Launch jobs locally via openjd CLI
Install with: pip install opencue-pyoutline[openjd]

Here's an example generated template based on the tutorials:

parallel_pipeline.template.yaml

linux-foundation-easycla · 2026-02-20T23:40:33Z

The committers listed above are authorized under a signed CLA.

✅ login: mwiebe / name: Mark Wiebe (bc82053)

DiegoTavares

Great work so far. I'm changing the PR description to add a checklist enlisting all features that are pending so we can keep tab on the status on the OpenJD project and even discuss what's essential.

DiegoTavares · 2026-02-25T18:26:49Z

pyoutline/outline/backend/openjd.py

+  - Layer outputs: the cue backend (spec >= 1.15) serializes get_outputs().
+    Not yet mapped.  OpenJD PATH parameters support dataFlow but only at
+    the job level, not per-step or per-task.
+    TODO: open a discussion on openjd-specifications for step- and task-level
+    file dataflow declarations (output path registration, existence checks).


This is crucial for some pipelines. Being able to trace what output was produced by each layer gives other systems the ability to trace assets for example for backups and recovers.

Agreed, another reason is to be more efficient about data transfers or taking snapshots of just what's needed to run a particular task.

DiegoTavares · 2026-02-25T18:35:14Z

pyoutline/outline/backend/openjd.py

+    file dataflow declarations (output path registration, existence checks).
+  - Launcher flags: pause, priority, maxretries, autoeat, facility, show,
+    shot, user, email, uid are OpenCue scheduling metadata with no direct
+    OpenJD equivalent.  The os flag could map to attr.worker.os.family.


Is there an equivalent way to launch a job as paused?

OpenJD doesn't have that concept, we modeled that in Deadline Cloud as part of a job that is running, while the job template at this level is concerned with defining the job's parallel structure and the commands it will run.

I think a number of these flags could map to job parameter name conventions, e.g. job parameters named Facility, Show, Shot, User, Email, Uid.

In my latest update, I've migrated a number of those to job parameters.

ramonfigueiredo · 2026-02-27T02:57:35Z

Hi @mwiebe

It is missing the CLA Authorization.

Thanks!

ramonfigueiredo · 2026-02-27T03:09:15Z

Please note that there are some Python linting issues that need to be addressed.

Additionally, when possible, remember to update the @docs/ for the new feature regarding the new feature in this pull request.

ramonfigueiredo

Thanks for your contribution, @mwiebe

I've added some comments and suggestions for code improvements.

ramonfigueiredo · 2026-02-27T03:14:22Z

pyoutline/outline/backend/openjd.py

+            attributes.append({"name": "opencue:attr.tag", "allOf": tag_list})
+
+    # Service → custom attribute opencue:attr.service
+    service = layer.get_service()


Service name not split on comma

The cue backend does layer.get_service().split(",")[0] to extract just the first service when multiple are comma-separated. The OpenJD backend passes the full string as-is:

service = layer.get_service() if service: attributes.append({"name": "opencue:attr.service", "anyOf": [service]})

If a layer has service="prman,shell", this would create anyOf: ["prman,shell"] instead of anyOf: ["prman"] (matching cue) or anyOf: ["prman", "shell"] (fully expanded). Should be intentional one way or the other.

I switched it to match the cue backend's behavior, with a comment about considering the expanded anyOf case. Would be useful to work through some examples of how to map this in practice and see what makes the best sense.

ramonfigueiredo · 2026-02-27T03:17:06Z

pyoutline/outline/backend/openjd.py

+        cmd = ["openjd", "run", tmp.name]
+
+        logger.info("Running: %s", " ".join(cmd))
+        result = subprocess.run(cmd, check=False)


launch() doesn't capture subprocess output

result = subprocess.run(cmd, check=False)

If openjd run fails, the error is just "openjd run exited with code N" with no diagnostic output. Consider capturing stderr:

result = subprocess.run(cmd, check=False, capture_output=True, text=True) if result.returncode != 0: raise outline.exception.OutlineException( "openjd run exited with code %d: %s" % (result.returncode, result.stderr) )

I've got the code following this pattern, but I still need to exercise it end to end to see the diagnostics working as expected.

pyoutline/outline/backend/openjd.py

ramonfigueiredo · 2026-02-27T04:19:39Z

pyoutline/outline/backend/openjd.py

+        # runs before setup.
+        # TODO: add a public get_code() method to PyEval so backends don't
+        # need to access the name-mangled attribute.
+        code = layer._PyEval__code


Private attribute access for PyEval

code = layer._PyEval__code

Acknowledged with a TODO. This is fragile, if the PyEval class renames or restructures its internals, this will break silently. Since this is a prototype, fine, but this should be addressed before any production use.

I can update the PyEval class as part of this PR.

ramonfigueiredo · 2026-02-27T04:21:22Z

pyoutline/pyproject.toml

@@ -1 +1 @@
 [build-system]


yaml (PyYAML) is not an explicit dependency:

The module imports yaml at the top level, but PyYAML isn't listed in pyproject.toml dependencies. It's likely pulled in transitively through opencue_pycue, but for correctness the dependency should be explicit, or at minimum noted.

I've added the yaml dependency. I'm also thinking to put openjd-model directly as a dependency instead of having a separate feature for now, to keep things simpler. When we've exercised the change more, let's discuss how we feel about the best way to incorporate it.

pyoutline/outline/backend/openjd.py

mwiebe · 2026-03-04T17:00:45Z

Hi @mwiebe

It is missing the CLA Authorization.

Thanks!

This is signed now, thanks!

Add a new PyOutline backend that serializes Outline jobs into OpenJD 2023-09 job templates. This enables running OpenCue jobs via the openjd CLI or submitting them to OpenJD-compatible schedulers like AWS Deadline Cloud. - Serialize layers to OpenJD steps with CHUNK[INT] frame parameters - Map OpenCue command tokens (#IFRAME#, #FRAMESPEC#, etc.) to OpenJD expressions using the EXPR extension - Map host requirements: cores, memory, GPUs, tags, service, limits - Map dependencies including Post layer implicit depends - Wrap commands in bash scripts that export CUE_* environment variables - Support Shell, ShellScript, and PyEval layer types - Validate generated templates using openjd-model - Launch jobs locally via openjd CLI - Install with: pip install opencue-pyoutline[openjd]

DiegoTavares reviewed Feb 25, 2026

View reviewed changes

ramonfigueiredo reviewed Feb 27, 2026

View reviewed changes

mwiebe force-pushed the openjd-prototype branch from efd7d4b to f252b29 Compare February 27, 2026 18:47

mwiebe force-pushed the openjd-prototype branch 6 times, most recently from 158cf35 to cd64582 Compare March 5, 2026 07:44

mwiebe force-pushed the openjd-prototype branch from cd64582 to bc82053 Compare March 5, 2026 07:45

Conversation

mwiebe commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linux-foundation-easycla bot commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DiegoTavares left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ramonfigueiredo commented Feb 27, 2026

Uh oh!

ramonfigueiredo commented Feb 27, 2026

Uh oh!

ramonfigueiredo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwiebe commented Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mwiebe commented Feb 20, 2026 •

edited

Loading

linux-foundation-easycla bot commented Feb 20, 2026 •

edited

Loading

DiegoTavares left a comment •

edited

Loading