Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
dc0d560
Draft JobOrchestrator ideas
SimonHeybrock Nov 15, 2025
e6efe7a
Context notes and thoughts
SimonHeybrock Nov 17, 2025
f7acccf
Implement JobOrchestrator for workflow lifecycle management
SimonHeybrock Nov 17, 2025
bfcaea9
Implement job stop logic in JobOrchestrator.commit_workflow
SimonHeybrock Nov 17, 2025
b0431bd
Move config persistence from WorkflowController to JobOrchestrator
SimonHeybrock Nov 17, 2025
beb7943
Clean up WorkflowController.start_workflow: remove dead code and simp…
SimonHeybrock Nov 17, 2025
95b019e
Fix bug and refactor JobOrchestrator to initialize all workflows with…
SimonHeybrock Nov 17, 2025
417f990
Fix JobOrchestrator to handle workflows with required params (correla…
SimonHeybrock Nov 17, 2025
581d728
Remove all workflow status methods and fields from WorkflowController…
SimonHeybrock Nov 17, 2025
b27850b
Refactor commit_workflow to return list[JobId] instead of JobNumber
SimonHeybrock Nov 17, 2025
ae85017
Refactor JobOrchestrator methods to use keyword-only arguments
SimonHeybrock Nov 17, 2025
6c5e092
Simplify JobOrchestrator by removing unnecessary Pydantic validation
SimonHeybrock Nov 17, 2025
3b159f0
Remove remaining pointless ConfigurationState validation in JobOrches…
SimonHeybrock Nov 17, 2025
be926e8
Remove unused methods from JobOrchestrator
SimonHeybrock Nov 17, 2025
57b02cd
Refactor JobSet to encapsulate job number generation and JobId creation.
SimonHeybrock Nov 17, 2025
e900e80
Simplify job_number assignment in commit_workflow
SimonHeybrock Nov 17, 2025
99e23e4
Refactor WorkflowConfig.from_params to only accept dicts
SimonHeybrock Nov 17, 2025
47c8654
Remove draft
SimonHeybrock Nov 17, 2025
22a69ec
Fix mutation safety issues in JobOrchestrator
SimonHeybrock Nov 17, 2025
d1b4133
Remove unused _workflow_specs_callbacks from WorkflowController
SimonHeybrock Nov 17, 2025
b3b75e0
Remove confusing docstring part
SimonHeybrock Nov 17, 2025
b005cf9
Remove unused status callback code from FakeWorkflowConfigService
SimonHeybrock Nov 17, 2025
c12cbeb
Remove unnecessary JobSet.add_job method
SimonHeybrock Nov 17, 2025
3c17c23
Comment and add more tests
SimonHeybrock Nov 17, 2025
fed2e2d
Fix YAML serialization error for enum values in JobOrchestrator
SimonHeybrock Nov 17, 2025
49a2201
Address PR #540 review comments
SimonHeybrock Nov 19, 2025
d96897c
Fix integration tests broken by source name validation
SimonHeybrock Nov 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 7 additions & 12 deletions src/ess/livedata/config/workflow_spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -272,22 +272,21 @@ class WorkflowConfig(BaseModel):
def from_params(
cls,
workflow_id: WorkflowId,
params: BaseModel | None = None,
aux_source_names: BaseModel | None = None,
params: dict | None = None,
aux_source_names: dict | None = None,
job_number: JobNumber | None = None,
) -> WorkflowConfig:
"""
Create a WorkflowConfig from validated Pydantic models.
Create a WorkflowConfig from parameters.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a note which briefly explains why the parameters are now just a dict and no longer Pydantic models? (Is it because dicts are easier to serialize if we want to persist them?)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The params are still a model in WorkflowSpec. The only difference is that the code that calls from_params now has a dict instead of a model (and might not know the model directly). I don't think it would be correct to explain this here, as it is just history.


Parameters
----------
workflow_id:
Identifier for the workflow
params:
Validated Pydantic model with workflow parameters, or None if no params
Workflow parameters as dict, or None if no params
aux_source_names:
Validated Pydantic model with auxiliary source selections, or None if no
aux sources
Auxiliary source selections as dict, or None if no aux sources
job_number:
Optional job number (generated if not provided)

Expand All @@ -299,12 +298,8 @@ def from_params(
return cls(
identifier=workflow_id,
job_number=job_number if job_number is not None else uuid.uuid4(),
aux_source_names=(
aux_source_names.model_dump(mode='json')
if aux_source_names is not None
else {}
),
params=params.model_dump() if params is not None else {},
aux_source_names=aux_source_names or {},
params=params or {},
)


Expand Down
17 changes: 17 additions & 0 deletions src/ess/livedata/dashboard/configuration_adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,23 @@ class ConfigurationState(BaseModel):
This model captures the user's configuration choices (sources, params,
aux sources) that should be restored when reopening the dashboard.
Used by both workflow and plotter configurations.

Schema Limitation
-----------------
This schema currently assumes all sources share the same `params` configuration,
with only `aux_source_names` varying per source. In reality, JobOrchestrator's
internal state (`staged_jobs`) allows different params per source via
`dict[SourceName, JobConfig]`.

For now, we expand on load: the single `params` dict is applied to all sources
in `source_names`, and `aux_source_names` is expanded per-source as needed.
This works because the current UI (WorkflowController.start_workflow) stages
the same params for all sources in a single operation.

Future work: If we support per-source params in the UI (e.g., "stage source1
with configA, stage source2 with configB"), this schema should be extended to:
`jobs: dict[str, JobConfigState]` where `JobConfigState` contains both params
and aux_source_names per source.
"""

source_names: list[str] = Field(
Expand Down
Loading
Loading